ampere arm tests

ARMv8 Neoverse-N1 testing with a System76 Thelio Astra (3.02 BIOS) and NVIDIA RTX A400/PCIe 4GB on Ubuntu 24.04 via the Phoronix Test Suite.

HTML result view exported from: https://openbenchmarking.org/result/2411249-PTS-AMPEREAR68&rdt&gru.

ampere arm testsProcessorMotherboardChipsetMemoryDiskGraphicsAudioMonitorNetworkOSKernelDesktopDisplay ServerDisplay DriverOpenGLCompilerFile-SystemScreen ResolutionabARMv8 Neoverse-N1 @ 3.00GHz (128 Cores)System76 Thelio Astra (3.02 BIOS)Ampere Computing LLC Altra PCI Root Complex A8 x 32GB DDR4-3200MT/s Micron 18ASF4G72PDZ-3G2F11024GB KINGSTON SKC3000S1024GNVIDIA RTX A400/PCIe 4GBNVIDIA Device 2291DELL P2415Q2 x Intel X550 + Intel I210Ubuntu 24.046.8.0-48-generic-64k (aarch64)GNOME Shell 46.0X ServerNVIDIA 550.1204.6.0GCC 13.2.0ext43840x2160OpenBenchmarking.orgKernel Details- Transparent Huge Pages: madviseCompiler Details- --build=aarch64-linux-gnu --disable-libquadmath --disable-libquadmath-support --disable-werror --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-fix-cortex-a53-843419 --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-backtrace --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-13-dIwDw0/gcc-13-13.2.0/debian/tmp-nvptx/usr --enable-plugin --enable-shared --enable-threads=posix --host=aarch64-linux-gnu --program-prefix=aarch64-linux-gnu- --target=aarch64-linux-gnu --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-target-system-zlib=auto --without-cuda-driver -v Processor Details- a: Scaling Governor: cppc_cpufreq schedutil (Boost: Disabled)- b: Scaling Governor: cppc_cpufreq performance (Boost: Disabled)Java Details- OpenJDK Runtime Environment (build 21.0.5+11-Ubuntu-1ubuntu124.04)Python Details- Python 3.12.3Security Details- gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + reg_file_data_sampling: Not affected + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of __user pointer sanitization + spectre_v2: Mitigation of CSV2 BHB + srbds: Not affected + tsx_async_abort: Not affected

ampere arm testsopenvino: Face Detection FP16 - CPUopenvino: Person Detection FP16 - CPUopenvino: Person Detection FP32 - CPUopenvino: Vehicle Detection FP16 - CPUopenvino: Face Detection FP16-INT8 - CPUopenvino: Face Detection Retail FP16 - CPUopenvino: Road Segmentation ADAS FP16 - CPUopenvino: Vehicle Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16 - CPUopenvino: Face Detection Retail FP16-INT8 - CPUopenvino: Road Segmentation ADAS FP16-INT8 - CPUopenvino: Machine Translation EN To DE FP16 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUopenvino: Person Vehicle Bike Detection FP16 - CPUopenvino: Noise Suppression Poconet-Like FP16 - CPUopenvino: Handwritten English Recognition FP16 - CPUopenvino: Person Re-Identification Retail FP16 - CPUopenvino: Age Gender Recognition Retail 0013 FP16 - CPUopenvino: Handwritten English Recognition FP16-INT8 - CPUopenvino: Age Gender Recognition Retail 0013 FP16-INT8 - CPUllama-cpp: CPU BLAS - Llama-3.1-Tulu-3-8B-Q8_0 - Text Generation 128llama-cpp: CPU BLAS - Llama-3.1-Tulu-3-8B-Q8_0 - Prompt Processing 512llama-cpp: CPU BLAS - Llama-3.1-Tulu-3-8B-Q8_0 - Prompt Processing 1024llama-cpp: CPU BLAS - Llama-3.1-Tulu-3-8B-Q8_0 - Prompt Processing 2048llama-cpp: CPU BLAS - Mistral-7B-Instruct-v0.3-Q8_0 - Text Generation 128llama-cpp: CPU BLAS - Mistral-7B-Instruct-v0.3-Q8_0 - Prompt Processing 512llama-cpp: CPU BLAS - Mistral-7B-Instruct-v0.3-Q8_0 - Prompt Processing 1024llama-cpp: CPU BLAS - Mistral-7B-Instruct-v0.3-Q8_0 - Prompt Processing 2048llama-cpp: CPU BLAS - granite-3.0-3b-a800m-instruct-Q8_0 - Text Generation 128llama-cpp: CPU BLAS - granite-3.0-3b-a800m-instruct-Q8_0 - Prompt Processing 512llama-cpp: CPU BLAS - granite-3.0-3b-a800m-instruct-Q8_0 - Prompt Processing 1024llama-cpp: CPU BLAS - granite-3.0-3b-a800m-instruct-Q8_0 - Prompt Processing 2048openvino-genai: Gemma-7b-int4-ov - CPUopenvino-genai: TinyLlama-1.1B-Chat-v1.0 - CPUopenvino-genai: Falcon-7b-instruct-int4-ov - CPUopenvino-genai: Phi-3-mini-128k-instruct-int4-ov - CPUrenaissance: Scala Dottyrenaissance: Rand Forestrenaissance: ALS Movie Lensrenaissance: Apache Spark Bayesrenaissance: Savina Reactors.IOrenaissance: Apache Spark PageRankrenaissance: Finagle HTTP Requestsrenaissance: Gaussian Mixture Modelrenaissance: In-Memory Database Shootoutrenaissance: Akka Unbalanced Cobwebbed Treerenaissance: Genetic Algorithm Using Jenetics + Futuresopenvino: Face Detection FP16 - CPUopenvino: Person Detection FP16 - CPUopenvino: Person Detection FP32 - CPUopenvino: Vehicle Detection FP16 - CPUopenvino: Face Detection FP16-INT8 - CPUopenvino: Face Detection Retail FP16 - CPUopenvino: Road Segmentation ADAS FP16 - CPUopenvino: Vehicle Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16 - CPUopenvino: Face Detection Retail FP16-INT8 - CPUopenvino: Road Segmentation ADAS FP16-INT8 - CPUopenvino: Machine Translation EN To DE FP16 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUopenvino: Person Vehicle Bike Detection FP16 - CPUopenvino: Noise Suppression Poconet-Like FP16 - CPUopenvino: Handwritten English Recognition FP16 - CPUopenvino: Person Re-Identification Retail FP16 - CPUopenvino: Age Gender Recognition Retail 0013 FP16 - CPUopenvino: Handwritten English Recognition FP16-INT8 - CPUopenvino: Age Gender Recognition Retail 0013 FP16-INT8 - CPUopenvino-genai: Gemma-7b-int4-ov - CPU - Time To First Tokenopenvino-genai: Gemma-7b-int4-ov - CPU - Time Per Output Tokenopenvino-genai: TinyLlama-1.1B-Chat-v1.0 - CPU - Time To First Tokenopenvino-genai: TinyLlama-1.1B-Chat-v1.0 - CPU - Time Per Output Tokenopenvino-genai: Falcon-7b-instruct-int4-ov - CPU - Time To First Tokenopenvino-genai: Falcon-7b-instruct-int4-ov - CPU - Time Per Output Tokenopenvino-genai: Phi-3-mini-128k-instruct-int4-ov - CPU - Time To First Tokenopenvino-genai: Phi-3-mini-128k-instruct-int4-ov - CPU - Time Per Output Tokenprimesieve: 1e12primesieve: 1e13ab6.8934.0333.99524.996.531224.24230.5286.55506.71392.4631.09299.07521.97297.9891.23177.16179.011848.45153.151850.0219.0496.79103.45104.8415.3896.79101.01105.4536.58199.65239.94261.25.9218.149.889.581360.1905.421263.0331.411584.03933.26084.36255.514065.843100.52583.24567.63937.32938.560.94788.6126.12138.6368.9563.1181.421021.92106.8661.26107.33350.23180.27178.6317.3208.5117.28201.31168.9882.4355.13121.27101.2131.88104.372.57541.0486.8933.9734.01549.616.531154.89232.286.94506.24392.9831.1300.92525.64294.3591.86177.13181.871818.46153.231970.511994.36103.7104.0216.2496.9103.43105.1638.61194.76229.86260.036.2320.919.869.661318.9907.620800.9330.711761.53959.65918.86162.514026.944671.62587.84571.1938.79937.7758.174793.1227.69137.56367.363.1781.321021.49106.2160.83108.65347.78180.37175.8317.59208.3716.23184.88160.4768.6547.83122.07101.42131.64103.512.59842.813OpenBenchmarking.org

OpenVINO

Model: Face Detection FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Face Detection FP16 - Device: CPUab2468106.896.891. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Person Detection FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Person Detection FP16 - Device: CPUab81624324034.0333.971. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Person Detection FP32 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Person Detection FP32 - Device: CPUab81624324033.9934.011. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Vehicle Detection FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Vehicle Detection FP16 - Device: CPUab120240360480600524.99549.611. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Face Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Face Detection FP16-INT8 - Device: CPUab2468106.536.531. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Face Detection Retail FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Face Detection Retail FP16 - Device: CPUab300600900120015001224.241154.891. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Road Segmentation ADAS FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Road Segmentation ADAS FP16 - Device: CPUab50100150200250230.52232.201. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Vehicle Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Vehicle Detection FP16-INT8 - Device: CPUab2040608010086.5586.941. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Weld Porosity Detection FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Weld Porosity Detection FP16 - Device: CPUab110220330440550506.71506.241. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Face Detection Retail FP16-INT8 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Face Detection Retail FP16-INT8 - Device: CPUab90180270360450392.46392.981. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Road Segmentation ADAS FP16-INT8 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Road Segmentation ADAS FP16-INT8 - Device: CPUab71421283531.0931.101. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Machine Translation EN To DE FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Machine Translation EN To DE FP16 - Device: CPUab70140210280350299.07300.921. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Weld Porosity Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Weld Porosity Detection FP16-INT8 - Device: CPUab110220330440550521.97525.641. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Person Vehicle Bike Detection FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Person Vehicle Bike Detection FP16 - Device: CPUab60120180240300297.98294.351. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Noise Suppression Poconet-Like FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Noise Suppression Poconet-Like FP16 - Device: CPUab2040608010091.2391.861. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Handwritten English Recognition FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Handwritten English Recognition FP16 - Device: CPUab4080120160200177.16177.131. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Person Re-Identification Retail FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Person Re-Identification Retail FP16 - Device: CPUab4080120160200179.01181.871. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Age Gender Recognition Retail 0013 FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Age Gender Recognition Retail 0013 FP16 - Device: CPUab4008001200160020001848.451818.461. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Handwritten English Recognition FP16-INT8 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Handwritten English Recognition FP16-INT8 - Device: CPUab306090120150153.15153.231. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Age Gender Recognition Retail 0013 FP16-INT8 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2024.5Model: Age Gender Recognition Retail 0013 FP16-INT8 - Device: CPUab4008001200160020001850.021970.511. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

Llama.cpp

Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Text Generation 128

OpenBenchmarking.orgTokens Per Second, More Is BetterLlama.cpp b4154Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Text Generation 128ab51015202519.0419.001. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -mcpu=native -fopenmp -lopenblas

Llama.cpp

Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 512

OpenBenchmarking.orgTokens Per Second, More Is BetterLlama.cpp b4154Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 512ab2040608010096.7994.361. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -mcpu=native -fopenmp -lopenblas

Llama.cpp

Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 1024

OpenBenchmarking.orgTokens Per Second, More Is BetterLlama.cpp b4154Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 1024ab20406080100103.45103.701. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -mcpu=native -fopenmp -lopenblas

Llama.cpp

Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 2048

OpenBenchmarking.orgTokens Per Second, More Is BetterLlama.cpp b4154Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 2048ab20406080100104.84104.021. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -mcpu=native -fopenmp -lopenblas

Llama.cpp

Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Text Generation 128

OpenBenchmarking.orgTokens Per Second, More Is BetterLlama.cpp b4154Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Text Generation 128ab4812162015.3816.241. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -mcpu=native -fopenmp -lopenblas

Llama.cpp

Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 512

OpenBenchmarking.orgTokens Per Second, More Is BetterLlama.cpp b4154Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 512ab2040608010096.7996.901. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -mcpu=native -fopenmp -lopenblas

Llama.cpp

Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 1024

OpenBenchmarking.orgTokens Per Second, More Is BetterLlama.cpp b4154Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 1024ab20406080100101.01103.431. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -mcpu=native -fopenmp -lopenblas

Llama.cpp

Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 2048

OpenBenchmarking.orgTokens Per Second, More Is BetterLlama.cpp b4154Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 2048ab20406080100105.45105.161. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -mcpu=native -fopenmp -lopenblas

Llama.cpp

Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Text Generation 128

OpenBenchmarking.orgTokens Per Second, More Is BetterLlama.cpp b4154Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Text Generation 128ab91827364536.5838.611. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -mcpu=native -fopenmp -lopenblas

Llama.cpp

Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 512

OpenBenchmarking.orgTokens Per Second, More Is BetterLlama.cpp b4154Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 512ab4080120160200199.65194.761. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -mcpu=native -fopenmp -lopenblas

Llama.cpp

Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 1024

OpenBenchmarking.orgTokens Per Second, More Is BetterLlama.cpp b4154Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 1024ab50100150200250239.94229.861. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -mcpu=native -fopenmp -lopenblas

Llama.cpp

Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 2048

OpenBenchmarking.orgTokens Per Second, More Is BetterLlama.cpp b4154Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 2048ab60120180240300261.20260.031. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -mcpu=native -fopenmp -lopenblas

OpenVINO GenAI

Model: Gemma-7b-int4-ov - Device: CPU

OpenBenchmarking.orgtokens/s, More Is BetterOpenVINO GenAI 2024.5Model: Gemma-7b-int4-ov - Device: CPUab2468105.926.23

OpenVINO GenAI

Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPU

OpenBenchmarking.orgtokens/s, More Is BetterOpenVINO GenAI 2024.5Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPUab51015202518.1420.91

OpenVINO GenAI

Model: Falcon-7b-instruct-int4-ov - Device: CPU

OpenBenchmarking.orgtokens/s, More Is BetterOpenVINO GenAI 2024.5Model: Falcon-7b-instruct-int4-ov - Device: CPUab36912159.889.86

OpenVINO GenAI

Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU

OpenBenchmarking.orgtokens/s, More Is BetterOpenVINO GenAI 2024.5Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPUab36912159.589.66

Renaissance

Test: Scala Dotty

OpenBenchmarking.orgms, Fewer Is BetterRenaissance 0.16Test: Scala Dottyab300600900120015001360.11318.9MIN: 1126.97 / MAX: 2195.04MIN: 1098.06 / MAX: 2074.06

Renaissance

Test: Random Forest

OpenBenchmarking.orgms, Fewer Is BetterRenaissance 0.16Test: Random Forestab2004006008001000905.4907.6MIN: 756.48 / MAX: 982.42MIN: 753.44 / MAX: 984.59

Renaissance

Test: ALS Movie Lens

OpenBenchmarking.orgms, Fewer Is BetterRenaissance 0.16Test: ALS Movie Lensab5K10K15K20K25K21263.020800.9MIN: 20319.56 / MAX: 21299.7MIN: 20144.95 / MAX: 20800.92

Renaissance

Test: Apache Spark Bayes

OpenBenchmarking.orgms, Fewer Is BetterRenaissance 0.16Test: Apache Spark Bayesab70140210280350331.4330.7MIN: 300.99 / MAX: 409.98MIN: 299.76 / MAX: 409.61

Renaissance

Test: Savina Reactors.IO

OpenBenchmarking.orgms, Fewer Is BetterRenaissance 0.16Test: Savina Reactors.IOab3K6K9K12K15K11584.011761.5MIN: 10895.07 / MAX: 11896.07MIN: 11188.89 / MAX: 13121.97

Renaissance

Test: Apache Spark PageRank

OpenBenchmarking.orgms, Fewer Is BetterRenaissance 0.16Test: Apache Spark PageRankab80016002400320040003933.23959.6MIN: 3619.83 / MAX: 3933.23MIN: 3632.35 / MAX: 3959.61

Renaissance

Test: Finagle HTTP Requests

OpenBenchmarking.orgms, Fewer Is BetterRenaissance 0.16Test: Finagle HTTP Requestsab130026003900520065006084.35918.8MIN: 5660.68 / MAX: 6387.6MIN: 5597 / MAX: 6068.47

Renaissance

Test: Gaussian Mixture Model

OpenBenchmarking.orgms, Fewer Is BetterRenaissance 0.16Test: Gaussian Mixture Modelab130026003900520065006255.56162.5MIN: 6255.49 / MAX: 6996.44MIN: 6162.48 / MAX: 6944.07

Renaissance

Test: In-Memory Database Shootout

OpenBenchmarking.orgms, Fewer Is BetterRenaissance 0.16Test: In-Memory Database Shootoutab3K6K9K12K15K14065.814026.9MIN: 13793.23 / MAX: 15145.96MIN: 13770.63 / MAX: 14791.41

Renaissance

Test: Akka Unbalanced Cobwebbed Tree

OpenBenchmarking.orgms, Fewer Is BetterRenaissance 0.16Test: Akka Unbalanced Cobwebbed Treeab10K20K30K40K50K43100.544671.6MAX: 43839.69MIN: 44225.66 / MAX: 45576.96

Renaissance

Test: Genetic Algorithm Using Jenetics + Futures

OpenBenchmarking.orgms, Fewer Is BetterRenaissance 0.16Test: Genetic Algorithm Using Jenetics + Futuresab60012001800240030002583.22587.8MIN: 2024.35 / MAX: 2583.21MIN: 1997.15 / MAX: 2587.82

OpenVINO

Model: Face Detection FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Face Detection FP16 - Device: CPUab100020003000400050004567.634571.10MIN: 2840.63 / MAX: 12274.43MIN: 2793.73 / MAX: 12373.741. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Person Detection FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Person Detection FP16 - Device: CPUab2004006008001000937.32938.79MIN: 489.42 / MAX: 1170.93MIN: 526.21 / MAX: 1168.251. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Person Detection FP32 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Person Detection FP32 - Device: CPUab2004006008001000938.50937.77MIN: 455.19 / MAX: 1165.35MIN: 531.07 / MAX: 1163.541. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Vehicle Detection FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Vehicle Detection FP16 - Device: CPUab142842567060.9058.17MIN: 16.22 / MAX: 130.67MIN: 15.88 / MAX: 118.671. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Face Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Face Detection FP16-INT8 - Device: CPUab100020003000400050004788.614793.12MIN: 3508.77 / MAX: 13349.18MIN: 3477.12 / MAX: 13307.711. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Face Detection Retail FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Face Detection Retail FP16 - Device: CPUab71421283526.1227.69MIN: 6.29 / MAX: 82.72MIN: 6.19 / MAX: 78.841. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Road Segmentation ADAS FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Road Segmentation ADAS FP16 - Device: CPUab306090120150138.60137.56MIN: 68.04 / MAX: 255.32MIN: 71.32 / MAX: 276.491. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Vehicle Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Vehicle Detection FP16-INT8 - Device: CPUab80160240320400368.95367.30MIN: 242.56 / MAX: 426.71MIN: 258.19 / MAX: 427.481. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Weld Porosity Detection FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Weld Porosity Detection FP16 - Device: CPUab142842567063.1163.17MIN: 8.46 / MAX: 1254.27MIN: 8.62 / MAX: 1257.711. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Face Detection Retail FP16-INT8 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Face Detection Retail FP16-INT8 - Device: CPUab2040608010081.4281.32MIN: 68.58 / MAX: 111.7MIN: 69.6 / MAX: 110.261. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Road Segmentation ADAS FP16-INT8 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Road Segmentation ADAS FP16-INT8 - Device: CPUab20040060080010001021.921021.49MIN: 713.38 / MAX: 1114.63MIN: 705.35 / MAX: 1115.91. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Machine Translation EN To DE FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Machine Translation EN To DE FP16 - Device: CPUab20406080100106.86106.21MIN: 97.37 / MAX: 159.08MIN: 96.3 / MAX: 152.741. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Weld Porosity Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Weld Porosity Detection FP16-INT8 - Device: CPUab142842567061.2660.83MIN: 26.59 / MAX: 1137.75MIN: 27.04 / MAX: 1145.551. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Person Vehicle Bike Detection FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Person Vehicle Bike Detection FP16 - Device: CPUab20406080100107.33108.65MIN: 24.19 / MAX: 176.55MIN: 29.12 / MAX: 173.291. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Noise Suppression Poconet-Like FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Noise Suppression Poconet-Like FP16 - Device: CPUab80160240320400350.23347.78MIN: 100.1 / MAX: 702.59MIN: 97.47 / MAX: 710.931. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Handwritten English Recognition FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Handwritten English Recognition FP16 - Device: CPUab4080120160200180.27180.37MIN: 175.48 / MAX: 263.73MIN: 176.16 / MAX: 264.31. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Person Re-Identification Retail FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Person Re-Identification Retail FP16 - Device: CPUab4080120160200178.63175.83MIN: 18.26 / MAX: 231.24MIN: 16.88 / MAX: 235.431. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Age Gender Recognition Retail 0013 FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Age Gender Recognition Retail 0013 FP16 - Device: CPUab4812162017.3017.59MIN: 0.97 / MAX: 293.08MIN: 1.17 / MAX: 295.881. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Handwritten English Recognition FP16-INT8 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Handwritten English Recognition FP16-INT8 - Device: CPUab50100150200250208.51208.37MIN: 203.39 / MAX: 292.57MIN: 202.58 / MAX: 292.611. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO

Model: Age Gender Recognition Retail 0013 FP16-INT8 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2024.5Model: Age Gender Recognition Retail 0013 FP16-INT8 - Device: CPUab4812162017.2816.23MIN: 7.99 / MAX: 296.13MIN: 7.16 / MAX: 294.421. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -std=c++14 -fPIC -fvisibility=hidden -MD -MT -MF

OpenVINO GenAI

Model: Gemma-7b-int4-ov - Device: CPU - Time To First Token

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO GenAI 2024.5Model: Gemma-7b-int4-ov - Device: CPU - Time To First Tokenab4080120160200201.31184.88

OpenVINO GenAI

Model: Gemma-7b-int4-ov - Device: CPU - Time Per Output Token

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO GenAI 2024.5Model: Gemma-7b-int4-ov - Device: CPU - Time Per Output Tokenab4080120160200168.98160.47

OpenVINO GenAI

Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPU - Time To First Token

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO GenAI 2024.5Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPU - Time To First Tokenab2040608010082.4368.65

OpenVINO GenAI

Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPU - Time Per Output Token

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO GenAI 2024.5Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPU - Time Per Output Tokenab122436486055.1347.83

OpenVINO GenAI

Model: Falcon-7b-instruct-int4-ov - Device: CPU - Time To First Token

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO GenAI 2024.5Model: Falcon-7b-instruct-int4-ov - Device: CPU - Time To First Tokenab306090120150121.27122.07

OpenVINO GenAI

Model: Falcon-7b-instruct-int4-ov - Device: CPU - Time Per Output Token

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO GenAI 2024.5Model: Falcon-7b-instruct-int4-ov - Device: CPU - Time Per Output Tokenab20406080100101.20101.42

OpenVINO GenAI

Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU - Time To First Token

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO GenAI 2024.5Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU - Time To First Tokenab306090120150131.88131.64

OpenVINO GenAI

Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU - Time Per Output Token

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO GenAI 2024.5Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU - Time Per Output Tokenab20406080100104.37103.51

Primesieve

Length: 1e12

OpenBenchmarking.orgSeconds, Fewer Is BetterPrimesieve 12.6Length: 1e12ab0.58461.16921.75382.33842.9232.5752.5981. (CXX) g++ options: -O3

Primesieve

Length: 1e13

OpenBenchmarking.orgSeconds, Fewer Is BetterPrimesieve 12.6Length: 1e13ab102030405041.0542.811. (CXX) g++ options: -O3

Phoronix Test Suite v10.8.5