altra openvino

ARMv8 Neoverse-N1 testing with a System76 Thelio Astra (3.02 BIOS) and NVIDIA RTX A400/PCIe 4GB on Ubuntu 24.04 via the Phoronix Test Suite.

Compare your own system(s) to this result file with the Phoronix Test Suite by running the command: phoronix-test-suite benchmark 2411240-PTS-ALTRAOPE71
Jump To Table - Results

Statistics

Remove Outliers Before Calculating Averages

Graph Settings

Prefer Vertical Bar Graphs

Multi-Way Comparison

Condense Multi-Option Tests Into Single Result Graphs

Table

Show Detailed System Result Table

Run Management

Result
Identifier
Performance Per
Dollar
Date
Run
  Test
  Duration
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -
November 24
  5 Hours, 6 Minutes
Only show results matching title/arguments (delimit multiple options with a comma):
Do not show results matching title/arguments (delimit multiple options with a comma):


altra openvinoOpenBenchmarking.orgPhoronix Test SuiteARMv8 Neoverse-N1 @ 3.00GHz (128 Cores)System76 Thelio Astra (3.02 BIOS)Ampere Computing LLC Altra PCI Root Complex A8 x 32GB DDR4-3200MT/s Micron 18ASF4G72PDZ-3G2F11024GB KINGSTON SKC3000S1024GNVIDIA RTX A400/PCIe 4GBNVIDIA Device 2291DELL P2415Q2 x Intel X550 + Intel I210Ubuntu 24.046.8.0-48-generic-64k (aarch64)GNOME Shell 46.0X ServerNVIDIA 550.1204.6.0GCC 13.2.0ext43840x2160ProcessorMotherboardChipsetMemoryDiskGraphicsAudioMonitorNetworkOSKernelDesktopDisplay ServerDisplay DriverOpenGLCompilerFile-SystemScreen ResolutionAltra Openvino BenchmarksSystem Logs- Transparent Huge Pages: madvise- --build=aarch64-linux-gnu --disable-libquadmath --disable-libquadmath-support --disable-werror --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-fix-cortex-a53-843419 --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-backtrace --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-13-dIwDw0/gcc-13-13.2.0/debian/tmp-nvptx/usr --enable-plugin --enable-shared --enable-threads=posix --host=aarch64-linux-gnu --program-prefix=aarch64-linux-gnu- --target=aarch64-linux-gnu --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-target-system-zlib=auto --without-cuda-driver -v - Scaling Governor: cppc_cpufreq performance (Boost: Disabled)- Python 3.12.3- gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + reg_file_data_sampling: Not affected + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of __user pointer sanitization + spectre_v2: Mitigation of CSV2 BHB + srbds: Not affected + tsx_async_abort: Not affected

altra openvinoopenvino-genai: Phi-3-mini-128k-instruct-int4-ov - CPUopenvino-genai: Falcon-7b-instruct-int4-ov - CPUopenvino-genai: TinyLlama-1.1B-Chat-v1.0 - CPUopenvino-genai: Gemma-7b-int4-ov - CPUopenvino: Age Gender Recognition Retail 0013 FP16-INT8 - CPUopenvino: Age Gender Recognition Retail 0013 FP16-INT8 - CPUopenvino: Handwritten English Recognition FP16-INT8 - CPUopenvino: Handwritten English Recognition FP16-INT8 - CPUopenvino: Age Gender Recognition Retail 0013 FP16 - CPUopenvino: Age Gender Recognition Retail 0013 FP16 - CPUopenvino: Person Re-Identification Retail FP16 - CPUopenvino: Person Re-Identification Retail FP16 - CPUopenvino: Handwritten English Recognition FP16 - CPUopenvino: Handwritten English Recognition FP16 - CPUopenvino: Noise Suppression Poconet-Like FP16 - CPUopenvino: Noise Suppression Poconet-Like FP16 - CPUopenvino: Person Vehicle Bike Detection FP16 - CPUopenvino: Person Vehicle Bike Detection FP16 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUopenvino: Machine Translation EN To DE FP16 - CPUopenvino: Machine Translation EN To DE FP16 - CPUopenvino: Road Segmentation ADAS FP16-INT8 - CPUopenvino: Road Segmentation ADAS FP16-INT8 - CPUopenvino: Face Detection Retail FP16-INT8 - CPUopenvino: Face Detection Retail FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16 - CPUopenvino: Weld Porosity Detection FP16 - CPUopenvino: Vehicle Detection FP16-INT8 - CPUopenvino: Vehicle Detection FP16-INT8 - CPUopenvino: Road Segmentation ADAS FP16 - CPUopenvino: Road Segmentation ADAS FP16 - CPUopenvino: Face Detection FP16-INT8 - CPUopenvino: Face Detection FP16-INT8 - CPUopenvino: Vehicle Detection FP16 - CPUopenvino: Vehicle Detection FP16 - CPUopenvino: Person Detection FP32 - CPUopenvino: Person Detection FP32 - CPUopenvino: Person Detection FP16 - CPUopenvino: Person Detection FP16 - CPUopenvino: Face Detection FP16 - CPUopenvino: Face Detection FP16 - CPUopenvino-genai: Phi-3-mini-128k-instruct-int4-ov - CPU - Time Per Output Tokenopenvino-genai: Phi-3-mini-128k-instruct-int4-ov - CPU - Time To First Tokenopenvino-genai: Falcon-7b-instruct-int4-ov - CPU - Time Per Output Tokenopenvino-genai: Falcon-7b-instruct-int4-ov - CPU - Time To First Tokenopenvino-genai: TinyLlama-1.1B-Chat-v1.0 - CPU - Time Per Output Tokenopenvino-genai: TinyLlama-1.1B-Chat-v1.0 - CPU - Time To First Tokenopenvino-genai: Gemma-7b-int4-ov - CPU - Time Per Output Tokenopenvino-genai: Gemma-7b-int4-ov - CPU - Time To First Tokenopenvino: Face Detection Retail FP16 - CPUopenvino: Face Detection Retail FP16 - CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -10.109.8519.766.0017.011880.88214.64148.8417.551822.54178.76178.91184.02173.59352.0690.76108.51294.7561.36521.12106.44300.361024.2831.0481.47392.2664.03499.46370.0186.31137.21232.834789.336.5360.15531.69937.2534.03937.6434.024566.756.9099.0125.29101.57121.9450.7272.65166.58198.5227.941153.54OpenBenchmarking.org

OpenVINO GenAI

OpenBenchmarking.orgtokens/s, More Is BetterOpenVINO GenAI 2024.5Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -3691215SE +/- 0.04, N = 310.10

OpenBenchmarking.orgtokens/s, More Is BetterOpenVINO GenAI 2024.5Model: Falcon-7b-instruct-int4-ov - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -3691215SE +/- 0.03, N = 39.85

OpenBenchmarking.orgtokens/s, More Is BetterOpenVINO GenAI 2024.5Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -510152025SE +/- 0.25, N = 1519.76

OpenBenchmarking.orgtokens/s, More Is BetterOpenVINO GenAI 2024.5Model: Gemma-7b-int4-ov - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -246810SE +/- 0.04, N = 36.00

OpenVINO

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Age Gender Recognition Retail 0013 FP16-INT8 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -48121620SE +/- 0.23, N = 317.01MIN: 1.5 / MAX: 294.61. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Age Gender Recognition Retail 0013 FP16-INT8 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -400800120016002000SE +/- 25.61, N = 31880.881. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Handwritten English Recognition FP16-INT8 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -50100150200250SE +/- 2.74, N = 3214.64MIN: 206.81 / MAX: 304.421. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Handwritten English Recognition FP16-INT8 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -306090120150SE +/- 1.86, N = 3148.841. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Age Gender Recognition Retail 0013 FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -48121620SE +/- 0.23, N = 317.55MIN: 0.91 / MAX: 295.11. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Age Gender Recognition Retail 0013 FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -400800120016002000SE +/- 24.18, N = 31822.541. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Person Re-Identification Retail FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -4080120160200SE +/- 0.61, N = 3178.76MIN: 16.36 / MAX: 249.371. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Person Re-Identification Retail FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -4080120160200SE +/- 0.62, N = 3178.911. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Handwritten English Recognition FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -4080120160200SE +/- 0.12, N = 3184.02MIN: 179.6 / MAX: 276.191. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Handwritten English Recognition FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -4080120160200SE +/- 0.09, N = 3173.591. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Noise Suppression Poconet-Like FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -80160240320400SE +/- 0.91, N = 3352.06MIN: 97.37 / MAX: 723.011. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Noise Suppression Poconet-Like FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -20406080100SE +/- 0.23, N = 390.761. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Person Vehicle Bike Detection FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -20406080100SE +/- 0.38, N = 3108.51MIN: 23.78 / MAX: 188.921. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Person Vehicle Bike Detection FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -60120180240300SE +/- 1.02, N = 3294.751. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Weld Porosity Detection FP16-INT8 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -1428425670SE +/- 0.02, N = 361.36MIN: 26.53 / MAX: 1259.531. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Weld Porosity Detection FP16-INT8 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -110220330440550SE +/- 0.18, N = 3521.121. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Machine Translation EN To DE FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -20406080100SE +/- 0.37, N = 3106.44MIN: 96.69 / MAX: 176.181. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Machine Translation EN To DE FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -70140210280350SE +/- 1.00, N = 3300.361. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Road Segmentation ADAS FP16-INT8 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -2004006008001000SE +/- 0.56, N = 31024.28MIN: 720.37 / MAX: 1124.871. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Road Segmentation ADAS FP16-INT8 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -714212835SE +/- 0.01, N = 331.041. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Face Detection Retail FP16-INT8 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -20406080100SE +/- 0.12, N = 381.47MIN: 63.53 / MAX: 114.031. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Face Detection Retail FP16-INT8 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -90180270360450SE +/- 0.57, N = 3392.261. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Weld Porosity Detection FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -1428425670SE +/- 0.43, N = 364.03MIN: 8.55 / MAX: 1282.191. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Weld Porosity Detection FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -110220330440550SE +/- 3.35, N = 3499.461. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Vehicle Detection FP16-INT8 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -80160240320400SE +/- 1.26, N = 3370.01MIN: 247.37 / MAX: 435.921. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Vehicle Detection FP16-INT8 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -20406080100SE +/- 0.30, N = 386.311. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Road Segmentation ADAS FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -306090120150SE +/- 0.07, N = 3137.21MIN: 64.84 / MAX: 275.791. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Road Segmentation ADAS FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -50100150200250SE +/- 0.12, N = 3232.831. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Face Detection FP16-INT8 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -10002000300040005000SE +/- 2.68, N = 34789.33MIN: 3410.69 / MAX: 13276.721. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Face Detection FP16-INT8 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -246810SE +/- 0.00, N = 36.531. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Vehicle Detection FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -1326395265SE +/- 0.60, N = 560.15MIN: 16.01 / MAX: 147.71. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Vehicle Detection FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -110220330440550SE +/- 5.23, N = 5531.691. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Person Detection FP32 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -2004006008001000SE +/- 0.26, N = 3937.25MIN: 366.98 / MAX: 1169.471. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Person Detection FP32 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -816243240SE +/- 0.01, N = 334.031. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Person Detection FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -2004006008001000SE +/- 0.52, N = 3937.64MIN: 472.78 / MAX: 1164.551. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Person Detection FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -816243240SE +/- 0.02, N = 334.021. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Face Detection FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -10002000300040005000SE +/- 2.59, N = 34566.75MIN: 2624.73 / MAX: 12337.221. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Face Detection FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -246810SE +/- 0.01, N = 36.901. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Face Detection Retail FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -714212835SE +/- 0.68, N = 1527.94MIN: 6.03 / MAX: 100.211. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Face Detection Retail FP16 - Device: CPUARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -2004006008001000SE +/- 25.65, N = 151153.541. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

44 Results Shown

OpenVINO GenAI:
  Phi-3-mini-128k-instruct-int4-ov - CPU
  Falcon-7b-instruct-int4-ov - CPU
  TinyLlama-1.1B-Chat-v1.0 - CPU
  Gemma-7b-int4-ov - CPU
OpenVINO:
  Age Gender Recognition Retail 0013 FP16-INT8 - CPU:
    ms
    FPS
  Handwritten English Recognition FP16-INT8 - CPU:
    ms
    FPS
  Age Gender Recognition Retail 0013 FP16 - CPU:
    ms
    FPS
  Person Re-Identification Retail FP16 - CPU:
    ms
    FPS
  Handwritten English Recognition FP16 - CPU:
    ms
    FPS
  Noise Suppression Poconet-Like FP16 - CPU:
    ms
    FPS
  Person Vehicle Bike Detection FP16 - CPU:
    ms
    FPS
  Weld Porosity Detection FP16-INT8 - CPU:
    ms
    FPS
  Machine Translation EN To DE FP16 - CPU:
    ms
    FPS
  Road Segmentation ADAS FP16-INT8 - CPU:
    ms
    FPS
  Face Detection Retail FP16-INT8 - CPU:
    ms
    FPS
  Weld Porosity Detection FP16 - CPU:
    ms
    FPS
  Vehicle Detection FP16-INT8 - CPU:
    ms
    FPS
  Road Segmentation ADAS FP16 - CPU:
    ms
    FPS
  Face Detection FP16-INT8 - CPU:
    ms
    FPS
  Vehicle Detection FP16 - CPU:
    ms
    FPS
  Person Detection FP32 - CPU:
    ms
    FPS
  Person Detection FP16 - CPU:
    ms
    FPS
  Face Detection FP16 - CPU:
    ms
    FPS
  Face Detection Retail FP16 - CPU:
    ms
    FPS