AMD EPYC Turin AI/ML Tuning Guide AMD EPYC 9655P following AMD tuning guide for AI/ML workloads - https://www.amd.com/content/dam/amd/en/documents/epyc-technical-docs/tuning-guides/58467_amd-epyc-9005-tg-bios-and-workload.pdf Benchmarks by Michael Larabel for a future article.
HTML result view exported from: https://openbenchmarking.org/result/2411286-NE-AMDEPYCTU24&grw .
AMD EPYC Turin AI/ML Tuning Guide Processor Motherboard Chipset Memory Disk Graphics Network OS Kernel Desktop Display Server Compiler File-System Screen Resolution Stock AI/ML Tuning Recommendations AMD EPYC 9655P 96-Core @ 2.60GHz (96 Cores / 192 Threads) Supermicro Super Server H13SSL-N v1.01 (3.0 BIOS) AMD 1Ah 12 x 64GB DDR5-6000MT/s Micron MTC40F2046S1RC64BDY QSFF 3201GB Micron_7450_MTFDKCB3T2TFS ASPEED 2 x Broadcom NetXtreme BCM5720 PCIe Ubuntu 24.10 6.12.0-rc7-linux-pm-next-phx (x86_64) GNOME Shell 47.0 X Server GCC 14.2.0 ext4 1024x768 OpenBenchmarking.org Kernel Details - Transparent Huge Pages: madvise Compiler Details - --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2,rust --enable-libphobos-checking=release --enable-libstdcxx-backtrace --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-14-zdkDXv/gcc-14-14.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-14-zdkDXv/gcc-14-14.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details - Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xb002116 Python Details - Python 3.12.7 Security Details - gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + reg_file_data_sampling: Not affected + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS; IBPB: conditional; STIBP: always-on; RSB filling; PBRSB-eIBRS: Not affected; BHI: Not affected + srbds: Not affected + tsx_async_abort: Not affected
AMD EPYC Turin AI/ML Tuning Guide litert: Mobilenet Float litert: NASNet Mobile litert: SqueezeNet litert: Inception V4 openvino-genai: Phi-3-mini-128k-instruct-int4-ov - CPU openvino-genai: Falcon-7b-instruct-int4-ov - CPU openvino-genai: Gemma-7b-int4-ov - CPU whisperfile: Tiny whisperfile: Small whisperfile: Medium tensorflow: CPU - 256 - ResNet-50 tensorflow: CPU - 512 - ResNet-50 numpy: onnx: ResNet50 v1-12-int8 - CPU - Standard onnx: ResNet101_DUC_HDC-12 - CPU - Standard pytorch: CPU - 256 - ResNet-50 pytorch: CPU - 256 - ResNet-152 pytorch: CPU - 512 - ResNet-50 pytorch: CPU - 512 - ResNet-152 whisper-cpp: ggml-small.en - 2016 State of the Union whisper-cpp: ggml-medium.en - 2016 State of the Union xnnpack: FP32MobileNetV1 xnnpack: FP32MobileNetV2 xnnpack: FP32MobileNetV3Small xnnpack: FP16MobileNetV1 xnnpack: FP16MobileNetV2 xnnpack: FP16MobileNetV3Small xnnpack: QS8MobileNetV2 llama-cpp: CPU BLAS - Llama-3.1-Tulu-3-8B-Q8_0 - Text Generation 128 llama-cpp: CPU BLAS - Llama-3.1-Tulu-3-8B-Q8_0 - Prompt Processing 512 llama-cpp: CPU BLAS - Llama-3.1-Tulu-3-8B-Q8_0 - Prompt Processing 1024 llama-cpp: CPU BLAS - Llama-3.1-Tulu-3-8B-Q8_0 - Prompt Processing 2048 llama-cpp: CPU BLAS - granite-3.0-3b-a800m-instruct-Q8_0 - Text Generation 128 llama-cpp: CPU BLAS - granite-3.0-3b-a800m-instruct-Q8_0 - Prompt Processing 512 llama-cpp: CPU BLAS - granite-3.0-3b-a800m-instruct-Q8_0 - Prompt Processing 2048 llama-cpp: CPU BLAS - Mistral-7B-Instruct-v0.3-Q8_0 - Text Generation 128 llama-cpp: CPU BLAS - Mistral-7B-Instruct-v0.3-Q8_0 - Prompt Processing 512 llama-cpp: CPU BLAS - Mistral-7B-Instruct-v0.3-Q8_0 - Prompt Processing 1024 llama-cpp: CPU BLAS - Mistral-7B-Instruct-v0.3-Q8_0 - Prompt Processing 2048 onednn: Convolution Batch Shapes Auto - CPU onednn: Deconvolution Batch shapes_1d - CPU onednn: Deconvolution Batch shapes_3d - CPU onednn: IP Shapes 1D - CPU onednn: IP Shapes 3D - CPU onednn: Recurrent Neural Network Training - CPU onednn: Recurrent Neural Network Inference - CPU openvino: Age Gender Recognition Retail 0013 FP16 - CPU openvino: Age Gender Recognition Retail 0013 FP16 - CPU openvino: Person Detection FP16 - CPU openvino: Person Detection FP16 - CPU openvino: Weld Porosity Detection FP16-INT8 - CPU openvino: Weld Porosity Detection FP16-INT8 - CPU openvino: Vehicle Detection FP16-INT8 - CPU openvino: Vehicle Detection FP16-INT8 - CPU openvino: Person Vehicle Bike Detection FP16 - CPU openvino: Person Vehicle Bike Detection FP16 - CPU openvino: Machine Translation EN To DE FP16 - CPU openvino: Machine Translation EN To DE FP16 - CPU openvino: Face Detection Retail FP16-INT8 - CPU openvino: Face Detection Retail FP16-INT8 - CPU openvino: Handwritten English Recognition FP16-INT8 - CPU openvino: Handwritten English Recognition FP16-INT8 - CPU openvino: Road Segmentation ADAS FP16-INT8 - CPU openvino: Road Segmentation ADAS FP16-INT8 - CPU openvino: Person Re-Identification Retail FP16 - CPU openvino: Person Re-Identification Retail FP16 - CPU openvino: Noise Suppression Poconet-Like FP16 - CPU openvino: Noise Suppression Poconet-Like FP16 - CPU openvino-genai: Phi-3-mini-128k-instruct-int4-ov - CPU - Time To First Token openvino-genai: Phi-3-mini-128k-instruct-int4-ov - CPU - Time Per Output Token openvino-genai: Falcon-7b-instruct-int4-ov - CPU - Time To First Token openvino-genai: Falcon-7b-instruct-int4-ov - CPU - Time Per Output Token openvino-genai: Gemma-7b-int4-ov - CPU - Time To First Token openvino-genai: Gemma-7b-int4-ov - CPU - Time Per Output Token onnx: ResNet50 v1-12-int8 - CPU - Standard onnx: ResNet101_DUC_HDC-12 - CPU - Standard Stock AI/ML Tuning Recommendations 4438.52 733737 7035.74 43898.9 55.63 51.01 37.84 31.88560 90.67175 200.48292 204.33 231.18 885.50 276.076 6.09788 51.61 20.78 51.48 20.60 221.62918 454.28794 4539 9203 10488 4634 9092 10966 10042 45.84 72.99 96.99 144.53 92.82 154.60 306.51 48.07 72.4 97.19 149.00 0.341475 6.70897 0.718484 0.535874 0.265564 425.718 276.379 140497.34 0.45 716.20 66.89 14035.76 6.72 8270.99 5.77 6720.98 7.09 852.83 56.24 23587.02 3.94 3559.87 26.94 2517.89 18.99 10525.86 4.53 6747.48 13.68 24.17 17.98 29.07 19.60 36.13 26.43 3.62341 165.094 4335.67 689396 6926.88 43824.9 56.46 51.12 38.00 31.43573 88.03640 197.24095 207.38 235.35 887.75 285.104 6.35626 53.13 21.79 53.34 21.71 214.62792 449.70262 4471 9062 10387 4513 9028 10323 9813 46.45 76.41 101.77 152.92 95.42 155.73 308.32 48.73 77.05 101.71 150.29 0.321833 6.65482 0.677050 0.507355 0.254123 406.453 262.517 146329.80 0.43 730.73 65.56 15437.79 6.09 8691.00 5.49 7144.84 6.66 881.22 54.43 25050.47 3.71 3649.63 26.28 2630.42 18.16 10800.12 4.41 6891.64 13.38 25.66 17.71 30.70 19.56 35.43 26.31 3.50697 158.528 OpenBenchmarking.org
LiteRT Model: Mobilenet Float OpenBenchmarking.org Microseconds, Fewer Is Better LiteRT 2024-10-15 Model: Mobilenet Float Stock AI/ML Tuning Recommendations 1000 2000 3000 4000 5000 SE +/- 11.59, N = 3 SE +/- 7.66, N = 3 4438.52 4335.67
LiteRT Model: NASNet Mobile OpenBenchmarking.org Microseconds, Fewer Is Better LiteRT 2024-10-15 Model: NASNet Mobile Stock AI/ML Tuning Recommendations 160K 320K 480K 640K 800K SE +/- 17324.48, N = 15 SE +/- 22050.37, N = 12 733737 689396
LiteRT Model: SqueezeNet OpenBenchmarking.org Microseconds, Fewer Is Better LiteRT 2024-10-15 Model: SqueezeNet Stock AI/ML Tuning Recommendations 1500 3000 4500 6000 7500 SE +/- 31.31, N = 3 SE +/- 31.55, N = 3 7035.74 6926.88
LiteRT Model: Inception V4 OpenBenchmarking.org Microseconds, Fewer Is Better LiteRT 2024-10-15 Model: Inception V4 Stock AI/ML Tuning Recommendations 9K 18K 27K 36K 45K SE +/- 47.06, N = 3 SE +/- 159.42, N = 3 43898.9 43824.9
OpenVINO GenAI Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU OpenBenchmarking.org tokens/s, More Is Better OpenVINO GenAI 2024.5 Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU Stock AI/ML Tuning Recommendations 13 26 39 52 65 SE +/- 0.16, N = 4 SE +/- 0.14, N = 4 55.63 56.46
OpenVINO GenAI Model: Falcon-7b-instruct-int4-ov - Device: CPU OpenBenchmarking.org tokens/s, More Is Better OpenVINO GenAI 2024.5 Model: Falcon-7b-instruct-int4-ov - Device: CPU Stock AI/ML Tuning Recommendations 12 24 36 48 60 SE +/- 0.09, N = 3 SE +/- 0.19, N = 3 51.01 51.12
OpenVINO GenAI Model: Gemma-7b-int4-ov - Device: CPU OpenBenchmarking.org tokens/s, More Is Better OpenVINO GenAI 2024.5 Model: Gemma-7b-int4-ov - Device: CPU Stock AI/ML Tuning Recommendations 9 18 27 36 45 SE +/- 0.13, N = 3 SE +/- 0.09, N = 3 37.84 38.00
Whisperfile Model Size: Tiny OpenBenchmarking.org Seconds, Fewer Is Better Whisperfile 20Aug24 Model Size: Tiny Stock AI/ML Tuning Recommendations 7 14 21 28 35 SE +/- 0.26, N = 3 SE +/- 0.25, N = 3 31.89 31.44
Whisperfile Model Size: Small OpenBenchmarking.org Seconds, Fewer Is Better Whisperfile 20Aug24 Model Size: Small Stock AI/ML Tuning Recommendations 20 40 60 80 100 SE +/- 0.71, N = 3 SE +/- 0.57, N = 3 90.67 88.04
Whisperfile Model Size: Medium OpenBenchmarking.org Seconds, Fewer Is Better Whisperfile 20Aug24 Model Size: Medium Stock AI/ML Tuning Recommendations 40 80 120 160 200 SE +/- 0.88, N = 3 SE +/- 0.71, N = 3 200.48 197.24
TensorFlow Device: CPU - Batch Size: 256 - Model: ResNet-50 OpenBenchmarking.org images/sec, More Is Better TensorFlow 2.16.1 Device: CPU - Batch Size: 256 - Model: ResNet-50 Stock AI/ML Tuning Recommendations 50 100 150 200 250 SE +/- 0.57, N = 3 SE +/- 0.72, N = 3 204.33 207.38
TensorFlow Device: CPU - Batch Size: 512 - Model: ResNet-50 OpenBenchmarking.org images/sec, More Is Better TensorFlow 2.16.1 Device: CPU - Batch Size: 512 - Model: ResNet-50 Stock AI/ML Tuning Recommendations 50 100 150 200 250 SE +/- 0.28, N = 3 SE +/- 0.20, N = 3 231.18 235.35
Numpy Benchmark OpenBenchmarking.org Score, More Is Better Numpy Benchmark Stock AI/ML Tuning Recommendations 200 400 600 800 1000 SE +/- 1.94, N = 3 SE +/- 0.72, N = 3 885.50 887.75
ONNX Runtime Model: ResNet50 v1-12-int8 - Device: CPU - Executor: Standard OpenBenchmarking.org Inferences Per Second, More Is Better ONNX Runtime 1.19 Model: ResNet50 v1-12-int8 - Device: CPU - Executor: Standard Stock AI/ML Tuning Recommendations 60 120 180 240 300 SE +/- 2.53, N = 7 SE +/- 1.46, N = 3 276.08 285.10 1. (CXX) g++ options: -O3 -march=native -ffunction-sections -fdata-sections -mtune=native -flto=auto -fno-fat-lto-objects -ldl -lrt
ONNX Runtime Model: ResNet101_DUC_HDC-12 - Device: CPU - Executor: Standard OpenBenchmarking.org Inferences Per Second, More Is Better ONNX Runtime 1.19 Model: ResNet101_DUC_HDC-12 - Device: CPU - Executor: Standard Stock AI/ML Tuning Recommendations 2 4 6 8 10 SE +/- 0.13191, N = 15 SE +/- 0.14383, N = 15 6.09788 6.35626 1. (CXX) g++ options: -O3 -march=native -ffunction-sections -fdata-sections -mtune=native -flto=auto -fno-fat-lto-objects -ldl -lrt
PyTorch Device: CPU - Batch Size: 256 - Model: ResNet-50 OpenBenchmarking.org batches/sec, More Is Better PyTorch 2.2.1 Device: CPU - Batch Size: 256 - Model: ResNet-50 Stock AI/ML Tuning Recommendations 12 24 36 48 60 SE +/- 0.15, N = 3 SE +/- 0.28, N = 3 51.61 53.13 MIN: 45.56 / MAX: 52.57 MIN: 46.57 / MAX: 54.35
PyTorch Device: CPU - Batch Size: 256 - Model: ResNet-152 OpenBenchmarking.org batches/sec, More Is Better PyTorch 2.2.1 Device: CPU - Batch Size: 256 - Model: ResNet-152 Stock AI/ML Tuning Recommendations 5 10 15 20 25 SE +/- 0.04, N = 3 SE +/- 0.27, N = 3 20.78 21.79 MIN: 19.72 / MAX: 21.04 MIN: 20.38 / MAX: 22.54
PyTorch Device: CPU - Batch Size: 512 - Model: ResNet-50 OpenBenchmarking.org batches/sec, More Is Better PyTorch 2.2.1 Device: CPU - Batch Size: 512 - Model: ResNet-50 Stock AI/ML Tuning Recommendations 12 24 36 48 60 SE +/- 0.15, N = 3 SE +/- 0.26, N = 3 51.48 53.34 MIN: 46.04 / MAX: 52.41 MIN: 49 / MAX: 54.59
PyTorch Device: CPU - Batch Size: 512 - Model: ResNet-152 OpenBenchmarking.org batches/sec, More Is Better PyTorch 2.2.1 Device: CPU - Batch Size: 512 - Model: ResNet-152 Stock AI/ML Tuning Recommendations 5 10 15 20 25 SE +/- 0.10, N = 3 SE +/- 0.18, N = 3 20.60 21.71 MIN: 19.3 / MAX: 21.02 MIN: 20.13 / MAX: 22.23
Whisper.cpp Model: ggml-small.en - Input: 2016 State of the Union OpenBenchmarking.org Seconds, Fewer Is Better Whisper.cpp 1.6.2 Model: ggml-small.en - Input: 2016 State of the Union Stock AI/ML Tuning Recommendations 50 100 150 200 250 SE +/- 2.33, N = 3 SE +/- 0.68, N = 3 221.63 214.63 1. (CXX) g++ options: -O3 -std=c++11 -fPIC -pthread -msse3 -mssse3 -mavx -mf16c -mfma -mavx2 -mavx512f -mavx512cd -mavx512vl -mavx512dq -mavx512bw -mavx512vbmi -mavx512vnni
Whisper.cpp Model: ggml-medium.en - Input: 2016 State of the Union OpenBenchmarking.org Seconds, Fewer Is Better Whisper.cpp 1.6.2 Model: ggml-medium.en - Input: 2016 State of the Union Stock AI/ML Tuning Recommendations 100 200 300 400 500 SE +/- 1.45, N = 3 SE +/- 1.24, N = 3 454.29 449.70 1. (CXX) g++ options: -O3 -std=c++11 -fPIC -pthread -msse3 -mssse3 -mavx -mf16c -mfma -mavx2 -mavx512f -mavx512cd -mavx512vl -mavx512dq -mavx512bw -mavx512vbmi -mavx512vnni
XNNPACK Model: FP32MobileNetV1 OpenBenchmarking.org us, Fewer Is Better XNNPACK b7b048 Model: FP32MobileNetV1 Stock AI/ML Tuning Recommendations 1000 2000 3000 4000 5000 SE +/- 42.62, N = 3 SE +/- 54.67, N = 3 4539 4471 1. (CXX) g++ options: -O3 -lrt -lm
XNNPACK Model: FP32MobileNetV2 OpenBenchmarking.org us, Fewer Is Better XNNPACK b7b048 Model: FP32MobileNetV2 Stock AI/ML Tuning Recommendations 2K 4K 6K 8K 10K SE +/- 32.13, N = 3 SE +/- 25.31, N = 3 9203 9062 1. (CXX) g++ options: -O3 -lrt -lm
XNNPACK Model: FP32MobileNetV3Small OpenBenchmarking.org us, Fewer Is Better XNNPACK b7b048 Model: FP32MobileNetV3Small Stock AI/ML Tuning Recommendations 2K 4K 6K 8K 10K SE +/- 11.35, N = 3 SE +/- 28.22, N = 3 10488 10387 1. (CXX) g++ options: -O3 -lrt -lm
XNNPACK Model: FP16MobileNetV1 OpenBenchmarking.org us, Fewer Is Better XNNPACK b7b048 Model: FP16MobileNetV1 Stock AI/ML Tuning Recommendations 1000 2000 3000 4000 5000 SE +/- 15.14, N = 3 SE +/- 10.73, N = 3 4634 4513 1. (CXX) g++ options: -O3 -lrt -lm
XNNPACK Model: FP16MobileNetV2 OpenBenchmarking.org us, Fewer Is Better XNNPACK b7b048 Model: FP16MobileNetV2 Stock AI/ML Tuning Recommendations 2K 4K 6K 8K 10K SE +/- 89.20, N = 3 SE +/- 94.57, N = 3 9092 9028 1. (CXX) g++ options: -O3 -lrt -lm
XNNPACK Model: FP16MobileNetV3Small OpenBenchmarking.org us, Fewer Is Better XNNPACK b7b048 Model: FP16MobileNetV3Small Stock AI/ML Tuning Recommendations 2K 4K 6K 8K 10K SE +/- 410.82, N = 3 SE +/- 21.33, N = 3 10966 10323 1. (CXX) g++ options: -O3 -lrt -lm
XNNPACK Model: QS8MobileNetV2 OpenBenchmarking.org us, Fewer Is Better XNNPACK b7b048 Model: QS8MobileNetV2 Stock AI/ML Tuning Recommendations 2K 4K 6K 8K 10K SE +/- 154.48, N = 3 SE +/- 87.21, N = 3 10042 9813 1. (CXX) g++ options: -O3 -lrt -lm
Llama.cpp Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Text Generation 128 OpenBenchmarking.org Tokens Per Second, More Is Better Llama.cpp b4154 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Text Generation 128 Stock AI/ML Tuning Recommendations 11 22 33 44 55 SE +/- 0.05, N = 4 SE +/- 0.09, N = 4 45.84 46.45 1. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -fopenmp -march=native -mtune=native -lopenblas
Llama.cpp Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 512 OpenBenchmarking.org Tokens Per Second, More Is Better Llama.cpp b4154 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 512 Stock AI/ML Tuning Recommendations 20 40 60 80 100 SE +/- 0.98, N = 3 SE +/- 0.99, N = 3 72.99 76.41 1. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -fopenmp -march=native -mtune=native -lopenblas
Llama.cpp Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 1024 OpenBenchmarking.org Tokens Per Second, More Is Better Llama.cpp b4154 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 1024 Stock AI/ML Tuning Recommendations 20 40 60 80 100 SE +/- 0.34, N = 3 SE +/- 1.16, N = 3 96.99 101.77 1. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -fopenmp -march=native -mtune=native -lopenblas
Llama.cpp Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 2048 OpenBenchmarking.org Tokens Per Second, More Is Better Llama.cpp b4154 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 2048 Stock AI/ML Tuning Recommendations 30 60 90 120 150 SE +/- 0.97, N = 3 SE +/- 1.38, N = 3 144.53 152.92 1. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -fopenmp -march=native -mtune=native -lopenblas
Llama.cpp Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Text Generation 128 OpenBenchmarking.org Tokens Per Second, More Is Better Llama.cpp b4154 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Text Generation 128 Stock AI/ML Tuning Recommendations 20 40 60 80 100 SE +/- 0.49, N = 6 SE +/- 0.43, N = 6 92.82 95.42 1. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -fopenmp -march=native -mtune=native -lopenblas
Llama.cpp Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 512 OpenBenchmarking.org Tokens Per Second, More Is Better Llama.cpp b4154 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 512 Stock AI/ML Tuning Recommendations 30 60 90 120 150 SE +/- 2.60, N = 12 SE +/- 3.23, N = 12 154.60 155.73 1. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -fopenmp -march=native -mtune=native -lopenblas
Llama.cpp Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 2048 OpenBenchmarking.org Tokens Per Second, More Is Better Llama.cpp b4154 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 2048 Stock AI/ML Tuning Recommendations 70 140 210 280 350 SE +/- 2.62, N = 3 SE +/- 2.23, N = 15 306.51 308.32 1. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -fopenmp -march=native -mtune=native -lopenblas
Llama.cpp Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Text Generation 128 OpenBenchmarking.org Tokens Per Second, More Is Better Llama.cpp b4154 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Text Generation 128 Stock AI/ML Tuning Recommendations 11 22 33 44 55 SE +/- 0.07, N = 4 SE +/- 0.05, N = 4 48.07 48.73 1. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -fopenmp -march=native -mtune=native -lopenblas
Llama.cpp Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 512 OpenBenchmarking.org Tokens Per Second, More Is Better Llama.cpp b4154 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 512 Stock AI/ML Tuning Recommendations 20 40 60 80 100 SE +/- 0.83, N = 3 SE +/- 0.77, N = 5 72.40 77.05 1. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -fopenmp -march=native -mtune=native -lopenblas
Llama.cpp Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 1024 OpenBenchmarking.org Tokens Per Second, More Is Better Llama.cpp b4154 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 1024 Stock AI/ML Tuning Recommendations 20 40 60 80 100 SE +/- 1.13, N = 4 SE +/- 0.67, N = 15 97.19 101.71 1. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -fopenmp -march=native -mtune=native -lopenblas
Llama.cpp Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 2048 OpenBenchmarking.org Tokens Per Second, More Is Better Llama.cpp b4154 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 2048 Stock AI/ML Tuning Recommendations 30 60 90 120 150 SE +/- 2.06, N = 3 SE +/- 1.36, N = 15 149.00 150.29 1. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -fopenmp -march=native -mtune=native -lopenblas
oneDNN Harness: Convolution Batch Shapes Auto - Engine: CPU OpenBenchmarking.org ms, Fewer Is Better oneDNN 3.6 Harness: Convolution Batch Shapes Auto - Engine: CPU Stock AI/ML Tuning Recommendations 0.0768 0.1536 0.2304 0.3072 0.384 SE +/- 0.000295, N = 7 SE +/- 0.001024, N = 7 0.341475 0.321833 MIN: 0.32 MIN: 0.31 1. (CXX) g++ options: -O3 -march=native -fopenmp -msse4.1 -fPIC -fcf-protection=full -pie -ldl
oneDNN Harness: Deconvolution Batch shapes_1d - Engine: CPU OpenBenchmarking.org ms, Fewer Is Better oneDNN 3.6 Harness: Deconvolution Batch shapes_1d - Engine: CPU Stock AI/ML Tuning Recommendations 2 4 6 8 10 SE +/- 0.03430, N = 3 SE +/- 0.01789, N = 3 6.70897 6.65482 MIN: 6.07 MIN: 3.91 1. (CXX) g++ options: -O3 -march=native -fopenmp -msse4.1 -fPIC -fcf-protection=full -pie -ldl
oneDNN Harness: Deconvolution Batch shapes_3d - Engine: CPU OpenBenchmarking.org ms, Fewer Is Better oneDNN 3.6 Harness: Deconvolution Batch shapes_3d - Engine: CPU Stock AI/ML Tuning Recommendations 0.1617 0.3234 0.4851 0.6468 0.8085 SE +/- 0.001205, N = 9 SE +/- 0.000482, N = 9 0.718484 0.677050 MIN: 0.62 MIN: 0.58 1. (CXX) g++ options: -O3 -march=native -fopenmp -msse4.1 -fPIC -fcf-protection=full -pie -ldl
oneDNN Harness: IP Shapes 1D - Engine: CPU OpenBenchmarking.org ms, Fewer Is Better oneDNN 3.6 Harness: IP Shapes 1D - Engine: CPU Stock AI/ML Tuning Recommendations 0.1206 0.2412 0.3618 0.4824 0.603 SE +/- 0.001151, N = 4 SE +/- 0.001097, N = 4 0.535874 0.507355 MIN: 0.49 MIN: 0.46 1. (CXX) g++ options: -O3 -march=native -fopenmp -msse4.1 -fPIC -fcf-protection=full -pie -ldl
oneDNN Harness: IP Shapes 3D - Engine: CPU OpenBenchmarking.org ms, Fewer Is Better oneDNN 3.6 Harness: IP Shapes 3D - Engine: CPU Stock AI/ML Tuning Recommendations 0.0598 0.1196 0.1794 0.2392 0.299 SE +/- 0.000944, N = 5 SE +/- 0.000491, N = 5 0.265564 0.254123 MIN: 0.24 MIN: 0.24 1. (CXX) g++ options: -O3 -march=native -fopenmp -msse4.1 -fPIC -fcf-protection=full -pie -ldl
oneDNN Harness: Recurrent Neural Network Training - Engine: CPU OpenBenchmarking.org ms, Fewer Is Better oneDNN 3.6 Harness: Recurrent Neural Network Training - Engine: CPU Stock AI/ML Tuning Recommendations 90 180 270 360 450 SE +/- 0.52, N = 3 SE +/- 0.37, N = 3 425.72 406.45 MIN: 419.47 MIN: 400.13 1. (CXX) g++ options: -O3 -march=native -fopenmp -msse4.1 -fPIC -fcf-protection=full -pie -ldl
oneDNN Harness: Recurrent Neural Network Inference - Engine: CPU OpenBenchmarking.org ms, Fewer Is Better oneDNN 3.6 Harness: Recurrent Neural Network Inference - Engine: CPU Stock AI/ML Tuning Recommendations 60 120 180 240 300 SE +/- 0.68, N = 3 SE +/- 0.27, N = 3 276.38 262.52 MIN: 269.7 MIN: 257.81 1. (CXX) g++ options: -O3 -march=native -fopenmp -msse4.1 -fPIC -fcf-protection=full -pie -ldl
OpenVINO Model: Age Gender Recognition Retail 0013 FP16 - Device: CPU OpenBenchmarking.org FPS, More Is Better OpenVINO 2024.5 Model: Age Gender Recognition Retail 0013 FP16 - Device: CPU Stock AI/ML Tuning Recommendations 30K 60K 90K 120K 150K SE +/- 313.81, N = 3 SE +/- 528.15, N = 3 140497.34 146329.80 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO Model: Age Gender Recognition Retail 0013 FP16 - Device: CPU OpenBenchmarking.org ms, Fewer Is Better OpenVINO 2024.5 Model: Age Gender Recognition Retail 0013 FP16 - Device: CPU Stock AI/ML Tuning Recommendations 0.1013 0.2026 0.3039 0.4052 0.5065 SE +/- 0.01, N = 3 SE +/- 0.00, N = 3 0.45 0.43 MIN: 0.16 / MAX: 25.14 MIN: 0.15 / MAX: 23.94 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO Model: Person Detection FP16 - Device: CPU OpenBenchmarking.org FPS, More Is Better OpenVINO 2024.5 Model: Person Detection FP16 - Device: CPU Stock AI/ML Tuning Recommendations 160 320 480 640 800 SE +/- 0.66, N = 3 SE +/- 0.74, N = 3 716.20 730.73 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO Model: Person Detection FP16 - Device: CPU OpenBenchmarking.org ms, Fewer Is Better OpenVINO 2024.5 Model: Person Detection FP16 - Device: CPU Stock AI/ML Tuning Recommendations 15 30 45 60 75 SE +/- 0.06, N = 3 SE +/- 0.07, N = 3 66.89 65.56 MIN: 34.58 / MAX: 130 MIN: 32.6 / MAX: 131.97 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO Model: Weld Porosity Detection FP16-INT8 - Device: CPU OpenBenchmarking.org FPS, More Is Better OpenVINO 2024.5 Model: Weld Porosity Detection FP16-INT8 - Device: CPU Stock AI/ML Tuning Recommendations 3K 6K 9K 12K 15K SE +/- 9.10, N = 3 SE +/- 14.35, N = 3 14035.76 15437.79 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO Model: Weld Porosity Detection FP16-INT8 - Device: CPU OpenBenchmarking.org ms, Fewer Is Better OpenVINO 2024.5 Model: Weld Porosity Detection FP16-INT8 - Device: CPU Stock AI/ML Tuning Recommendations 2 4 6 8 10 SE +/- 0.00, N = 3 SE +/- 0.00, N = 3 6.72 6.09 MIN: 2.22 / MAX: 22.18 MIN: 2.21 / MAX: 23.86 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO Model: Vehicle Detection FP16-INT8 - Device: CPU OpenBenchmarking.org FPS, More Is Better OpenVINO 2024.5 Model: Vehicle Detection FP16-INT8 - Device: CPU Stock AI/ML Tuning Recommendations 2K 4K 6K 8K 10K SE +/- 3.96, N = 3 SE +/- 7.22, N = 3 8270.99 8691.00 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO Model: Vehicle Detection FP16-INT8 - Device: CPU OpenBenchmarking.org ms, Fewer Is Better OpenVINO 2024.5 Model: Vehicle Detection FP16-INT8 - Device: CPU Stock AI/ML Tuning Recommendations 1.2983 2.5966 3.8949 5.1932 6.4915 SE +/- 0.00, N = 3 SE +/- 0.01, N = 3 5.77 5.49 MIN: 2.28 / MAX: 21.36 MIN: 2.47 / MAX: 19.56 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO Model: Person Vehicle Bike Detection FP16 - Device: CPU OpenBenchmarking.org FPS, More Is Better OpenVINO 2024.5 Model: Person Vehicle Bike Detection FP16 - Device: CPU Stock AI/ML Tuning Recommendations 1500 3000 4500 6000 7500 SE +/- 13.33, N = 3 SE +/- 8.11, N = 3 6720.98 7144.84 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO Model: Person Vehicle Bike Detection FP16 - Device: CPU OpenBenchmarking.org ms, Fewer Is Better OpenVINO 2024.5 Model: Person Vehicle Bike Detection FP16 - Device: CPU Stock AI/ML Tuning Recommendations 2 4 6 8 10 SE +/- 0.01, N = 3 SE +/- 0.01, N = 3 7.09 6.66 MIN: 4.15 / MAX: 20.16 MIN: 3.65 / MAX: 22.09 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO Model: Machine Translation EN To DE FP16 - Device: CPU OpenBenchmarking.org FPS, More Is Better OpenVINO 2024.5 Model: Machine Translation EN To DE FP16 - Device: CPU Stock AI/ML Tuning Recommendations 200 400 600 800 1000 SE +/- 0.22, N = 3 SE +/- 0.40, N = 3 852.83 881.22 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO Model: Machine Translation EN To DE FP16 - Device: CPU OpenBenchmarking.org ms, Fewer Is Better OpenVINO 2024.5 Model: Machine Translation EN To DE FP16 - Device: CPU Stock AI/ML Tuning Recommendations 13 26 39 52 65 SE +/- 0.02, N = 3 SE +/- 0.03, N = 3 56.24 54.43 MIN: 29.3 / MAX: 94.69 MIN: 28.33 / MAX: 92.29 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO Model: Face Detection Retail FP16-INT8 - Device: CPU OpenBenchmarking.org FPS, More Is Better OpenVINO 2024.5 Model: Face Detection Retail FP16-INT8 - Device: CPU Stock AI/ML Tuning Recommendations 5K 10K 15K 20K 25K SE +/- 17.66, N = 3 SE +/- 17.84, N = 3 23587.02 25050.47 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO Model: Face Detection Retail FP16-INT8 - Device: CPU OpenBenchmarking.org ms, Fewer Is Better OpenVINO 2024.5 Model: Face Detection Retail FP16-INT8 - Device: CPU Stock AI/ML Tuning Recommendations 0.8865 1.773 2.6595 3.546 4.4325 SE +/- 0.01, N = 3 SE +/- 0.00, N = 3 3.94 3.71 MIN: 1.76 / MAX: 17.65 MIN: 1.71 / MAX: 19.33 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO Model: Handwritten English Recognition FP16-INT8 - Device: CPU OpenBenchmarking.org FPS, More Is Better OpenVINO 2024.5 Model: Handwritten English Recognition FP16-INT8 - Device: CPU Stock AI/ML Tuning Recommendations 800 1600 2400 3200 4000 SE +/- 2.96, N = 3 SE +/- 3.35, N = 3 3559.87 3649.63 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO Model: Handwritten English Recognition FP16-INT8 - Device: CPU OpenBenchmarking.org ms, Fewer Is Better OpenVINO 2024.5 Model: Handwritten English Recognition FP16-INT8 - Device: CPU Stock AI/ML Tuning Recommendations 6 12 18 24 30 SE +/- 0.02, N = 3 SE +/- 0.02, N = 3 26.94 26.28 MIN: 15.65 / MAX: 45.15 MIN: 15.86 / MAX: 40.89 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO Model: Road Segmentation ADAS FP16-INT8 - Device: CPU OpenBenchmarking.org FPS, More Is Better OpenVINO 2024.5 Model: Road Segmentation ADAS FP16-INT8 - Device: CPU Stock AI/ML Tuning Recommendations 600 1200 1800 2400 3000 SE +/- 7.45, N = 3 SE +/- 2.42, N = 3 2517.89 2630.42 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO Model: Road Segmentation ADAS FP16-INT8 - Device: CPU OpenBenchmarking.org ms, Fewer Is Better OpenVINO 2024.5 Model: Road Segmentation ADAS FP16-INT8 - Device: CPU Stock AI/ML Tuning Recommendations 5 10 15 20 25 SE +/- 0.06, N = 3 SE +/- 0.02, N = 3 18.99 18.16 MIN: 9.19 / MAX: 39 MIN: 7.68 / MAX: 40.38 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO Model: Person Re-Identification Retail FP16 - Device: CPU OpenBenchmarking.org FPS, More Is Better OpenVINO 2024.5 Model: Person Re-Identification Retail FP16 - Device: CPU Stock AI/ML Tuning Recommendations 2K 4K 6K 8K 10K SE +/- 12.77, N = 3 SE +/- 6.08, N = 3 10525.86 10800.12 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO Model: Person Re-Identification Retail FP16 - Device: CPU OpenBenchmarking.org ms, Fewer Is Better OpenVINO 2024.5 Model: Person Re-Identification Retail FP16 - Device: CPU Stock AI/ML Tuning Recommendations 1.0193 2.0386 3.0579 4.0772 5.0965 SE +/- 0.01, N = 3 SE +/- 0.00, N = 3 4.53 4.41 MIN: 1.95 / MAX: 23.94 MIN: 2.45 / MAX: 17.46 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO Model: Noise Suppression Poconet-Like FP16 - Device: CPU OpenBenchmarking.org FPS, More Is Better OpenVINO 2024.5 Model: Noise Suppression Poconet-Like FP16 - Device: CPU Stock AI/ML Tuning Recommendations 1500 3000 4500 6000 7500 SE +/- 11.12, N = 3 SE +/- 9.90, N = 3 6747.48 6891.64 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO Model: Noise Suppression Poconet-Like FP16 - Device: CPU OpenBenchmarking.org ms, Fewer Is Better OpenVINO 2024.5 Model: Noise Suppression Poconet-Like FP16 - Device: CPU Stock AI/ML Tuning Recommendations 4 8 12 16 20 SE +/- 0.02, N = 3 SE +/- 0.02, N = 3 13.68 13.38 MIN: 7.09 / MAX: 36.01 MIN: 6.98 / MAX: 34.62 1. (CXX) g++ options: -fPIC -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -shared -ldl -lstdc++fs
OpenVINO CPU Power Consumption Monitor Min Avg Max Stock 1.5 262.5 291.6 AI/ML Tuning Recommendations 3.7 275.9 306.1 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 CPU Power Consumption Monitor 80 160 240 320 400
OpenVINO System Power Consumption Monitor Min Avg Max Stock 99.1 436.2 470.9 AI/ML Tuning Recommendations 100.1 457.0 491.2 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 System Power Consumption Monitor 130 260 390 520 650
OpenVINO CPU Power Consumption Monitor Min Avg Max Stock 62.9 324.7 356.2 AI/ML Tuning Recommendations 78.6 339.7 368.6 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 CPU Power Consumption Monitor 100 200 300 400 500
OpenVINO System Power Consumption Monitor Min Avg Max Stock 100 594 647 AI/ML Tuning Recommendations 100 615 672 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 System Power Consumption Monitor 200 400 600 800 1000
OpenVINO CPU Power Consumption Monitor Min Avg Max Stock 83.1 271.0 295.4 AI/ML Tuning Recommendations 94.6 304.8 328.6 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 CPU Power Consumption Monitor 80 160 240 320 400
OpenVINO System Power Consumption Monitor Min Avg Max Stock 102.3 440.1 472.4 AI/ML Tuning Recommendations 105.1 501.8 525.4 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 System Power Consumption Monitor 130 260 390 520 650
OpenVINO CPU Power Consumption Monitor Min Avg Max Stock 49.3 290.5 319.5 AI/ML Tuning Recommendations 132.1 314.2 336.5 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 CPU Power Consumption Monitor 80 160 240 320 400
OpenVINO System Power Consumption Monitor Min Avg Max Stock 100.0 485.3 521.8 AI/ML Tuning Recommendations 99.2 521.4 553.7 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 System Power Consumption Monitor 140 280 420 560 700
OpenVINO CPU Power Consumption Monitor Min Avg Max Stock 32.7 255.4 281.9 AI/ML Tuning Recommendations 79.1 276.2 299.8 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 CPU Power Consumption Monitor 80 160 240 320 400
OpenVINO System Power Consumption Monitor Min Avg Max Stock 99.3 441.8 469.2 AI/ML Tuning Recommendations 98.7 461.9 496.8 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 System Power Consumption Monitor 130 260 390 520 650
OpenVINO CPU Power Consumption Monitor Min Avg Max Stock 16.6 313.1 351.4 AI/ML Tuning Recommendations 79.4 334.8 365.0 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 CPU Power Consumption Monitor 100 200 300 400 500
OpenVINO System Power Consumption Monitor Min Avg Max Stock 100 556 612 AI/ML Tuning Recommendations 99 584 636 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 System Power Consumption Monitor 200 400 600 800 1000
OpenVINO CPU Power Consumption Monitor Min Avg Max Stock 13.5 282.0 312.1 AI/ML Tuning Recommendations 90.1 309.0 331.5 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 CPU Power Consumption Monitor 80 160 240 320 400
OpenVINO System Power Consumption Monitor Min Avg Max Stock 101.1 472.8 504.1 AI/ML Tuning Recommendations 101.2 511.1 536.6 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 System Power Consumption Monitor 140 280 420 560 700
OpenVINO CPU Power Consumption Monitor Min Avg Max Stock 6.9 317.3 351.6 AI/ML Tuning Recommendations 117.3 335.4 361.8 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 CPU Power Consumption Monitor 100 200 300 400 500
OpenVINO System Power Consumption Monitor Min Avg Max Stock 99.3 549.0 578.2 AI/ML Tuning Recommendations 100.2 560.5 599.6 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 System Power Consumption Monitor 160 320 480 640 800
OpenVINO CPU Power Consumption Monitor Min Avg Max Stock 5.3 278.3 306.1 AI/ML Tuning Recommendations 81.9 300.1 323.0 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 CPU Power Consumption Monitor 80 160 240 320 400
OpenVINO System Power Consumption Monitor Min Avg Max Stock 98.2 485.5 529.0 AI/ML Tuning Recommendations 101.1 512.1 561.7 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 System Power Consumption Monitor 140 280 420 560 700
OpenVINO CPU Power Consumption Monitor Min Avg Max Stock 0.2 301.5 333.3 AI/ML Tuning Recommendations 81.7 320.5 341.9 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 CPU Power Consumption Monitor 80 160 240 320 400
OpenVINO System Power Consumption Monitor Min Avg Max Stock 100.6 506.3 535.9 AI/ML Tuning Recommendations 100.0 523.2 551.8 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 System Power Consumption Monitor 140 280 420 560 700
OpenVINO CPU Power Consumption Monitor Min Avg Max Stock 26.6 283.4 308.8 AI/ML Tuning Recommendations 106.9 299.0 322.1 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 CPU Power Consumption Monitor 80 160 240 320 400
OpenVINO System Power Consumption Monitor Min Avg Max Stock 98.8 523.1 566.9 AI/ML Tuning Recommendations 99.0 545.2 588.7 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2024.5 System Power Consumption Monitor 160 320 480 640 800
OpenVINO GenAI CPU Power Consumption Monitor Min Avg Max Stock 0.8 190.8 280.5 AI/ML Tuning Recommendations 69.4 248.5 348.6 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO GenAI 2024.5 CPU Power Consumption Monitor 100 200 300 400 500
OpenVINO GenAI System Power Consumption Monitor Min Avg Max Stock 99.1 331.8 453.9 AI/ML Tuning Recommendations 98.1 380.9 543.2 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO GenAI 2024.5 System Power Consumption Monitor 140 280 420 560 700
OpenVINO GenAI CPU Power Consumption Monitor Min Avg Max Stock 6.1 239.0 309.8 AI/ML Tuning Recommendations 52.7 293.6 364.5 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO GenAI 2024.5 CPU Power Consumption Monitor 100 200 300 400 500
OpenVINO GenAI System Power Consumption Monitor Min Avg Max Stock 98.2 426.7 509.8 AI/ML Tuning Recommendations 98.1 490.1 585.6 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO GenAI 2024.5 System Power Consumption Monitor 160 320 480 640 800
OpenVINO GenAI CPU Power Consumption Monitor Min Avg Max Stock 14.0 235.6 293.0 AI/ML Tuning Recommendations 46.1 291.1 341.4 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO GenAI 2024.5 CPU Power Consumption Monitor 80 160 240 320 400
OpenVINO GenAI System Power Consumption Monitor Min Avg Max Stock 99.9 420.1 486.7 AI/ML Tuning Recommendations 97.9 483.3 551.8 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO GenAI 2024.5 System Power Consumption Monitor 140 280 420 560 700
Llama.cpp CPU Power Consumption Monitor Min Avg Max Stock 0.6 236.1 334.7 AI/ML Tuning Recommendations 57.2 321.1 401.3 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 CPU Power Consumption Monitor 110 220 330 440 550
Llama.cpp System Power Consumption Monitor Min Avg Max Stock 98 390 558 AI/ML Tuning Recommendations 99 536 650 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 System Power Consumption Monitor 200 400 600 800 1000
Llama.cpp CPU Power Consumption Monitor Min Avg Max Stock 7.3 221.9 263.5 AI/ML Tuning Recommendations 126.7 294.6 332.7 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 CPU Power Consumption Monitor 80 160 240 320 400
Llama.cpp System Power Consumption Monitor Min Avg Max Stock 98.4 365.8 443.9 AI/ML Tuning Recommendations 100.4 456.3 525.6 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 System Power Consumption Monitor 130 260 390 520 650
Llama.cpp CPU Power Consumption Monitor Min Avg Max Stock 17.5 231.5 260.6 AI/ML Tuning Recommendations 123.4 289.4 326.4 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 CPU Power Consumption Monitor 80 160 240 320 400
Llama.cpp System Power Consumption Monitor Min Avg Max Stock 98.5 382.8 438.4 AI/ML Tuning Recommendations 98.4 450.9 513.6 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 System Power Consumption Monitor 130 260 390 520 650
Llama.cpp CPU Power Consumption Monitor Min Avg Max Stock 11.3 247.5 289.5 AI/ML Tuning Recommendations 114.5 287.8 315.6 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 CPU Power Consumption Monitor 80 160 240 320 400
Llama.cpp System Power Consumption Monitor Min Avg Max Stock 97.9 410.1 502.2 AI/ML Tuning Recommendations 100.3 454.2 543.8 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 System Power Consumption Monitor 140 280 420 560 700
Llama.cpp CPU Power Consumption Monitor Min Avg Max Stock 11.3 165.3 269.9 AI/ML Tuning Recommendations 86.4 203.0 305.1 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 CPU Power Consumption Monitor 80 160 240 320 400
Llama.cpp System Power Consumption Monitor Min Avg Max Stock 96.7 256.1 410.9 AI/ML Tuning Recommendations 99.9 323.5 464.8 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 System Power Consumption Monitor 120 240 360 480 600
Llama.cpp CPU Power Consumption Monitor Min Avg Max Stock 14.5 244.4 300.1 AI/ML Tuning Recommendations 97.3 292.2 349.7 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 CPU Power Consumption Monitor 100 200 300 400 500
Llama.cpp System Power Consumption Monitor Min Avg Max Stock 97.9 371.5 450.0 AI/ML Tuning Recommendations 97.3 433.6 515.3 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 System Power Consumption Monitor 130 260 390 520 650
Llama.cpp CPU Power Consumption Monitor Min Avg Max Stock 12.7 254.5 302.2 AI/ML Tuning Recommendations 95.6 307.1 351.1 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 CPU Power Consumption Monitor 100 200 300 400 500
Llama.cpp System Power Consumption Monitor Min Avg Max Stock 98.1 411.1 499.8 AI/ML Tuning Recommendations 98.2 467.8 533.6 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 System Power Consumption Monitor 140 280 420 560 700
Llama.cpp CPU Power Consumption Monitor Min Avg Max Stock 6.0 238.9 331.9 AI/ML Tuning Recommendations 56.8 293.0 399.7 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 CPU Power Consumption Monitor 110 220 330 440 550
Llama.cpp System Power Consumption Monitor Min Avg Max Stock 98 428 552 AI/ML Tuning Recommendations 99 462 644 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 System Power Consumption Monitor 200 400 600 800 1000
Llama.cpp CPU Power Consumption Monitor Min Avg Max Stock 9.9 224.4 269.8 AI/ML Tuning Recommendations 94.7 289.4 335.4 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 CPU Power Consumption Monitor 80 160 240 320 400
Llama.cpp System Power Consumption Monitor Min Avg Max Stock 98.9 372.0 414.5 AI/ML Tuning Recommendations 100.3 442.0 512.9 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 System Power Consumption Monitor 130 260 390 520 650
Llama.cpp CPU Power Consumption Monitor Min Avg Max Stock 3.1 232.5 266.0 AI/ML Tuning Recommendations 92.1 285.4 337.6 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 CPU Power Consumption Monitor 80 160 240 320 400
Llama.cpp System Power Consumption Monitor Min Avg Max Stock 98.9 375.5 446.8 AI/ML Tuning Recommendations 98.9 444.6 524.3 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 System Power Consumption Monitor 130 260 390 520 650
Llama.cpp CPU Power Consumption Monitor Min Avg Max Stock 2.7 248.2 291.5 AI/ML Tuning Recommendations 86.4 288.4 323.2 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 CPU Power Consumption Monitor 80 160 240 320 400
Llama.cpp System Power Consumption Monitor Min Avg Max Stock 99.2 415.9 506.6 AI/ML Tuning Recommendations 98.9 463.4 547.0 OpenBenchmarking.org Watts, Fewer Is Better Llama.cpp b4154 System Power Consumption Monitor 140 280 420 560 700
Whisperfile CPU Power Consumption Monitor Min Avg Max Stock 2.7 137.0 193.3 AI/ML Tuning Recommendations 0.4 155.0 217.0 OpenBenchmarking.org Watts, Fewer Is Better Whisperfile 20Aug24 CPU Power Consumption Monitor 60 120 180 240 300
Whisperfile System Power Consumption Monitor Min Avg Max Stock 99.3 256.2 339.0 AI/ML Tuning Recommendations 98.6 285.0 376.0 OpenBenchmarking.org Watts, Fewer Is Better Whisperfile 20Aug24 System Power Consumption Monitor 100 200 300 400 500
Whisperfile CPU Power Consumption Monitor Min Avg Max Stock 5.5 180.6 208.1 AI/ML Tuning Recommendations 46.8 200.1 231.1 OpenBenchmarking.org Watts, Fewer Is Better Whisperfile 20Aug24 CPU Power Consumption Monitor 60 120 180 240 300
Whisperfile System Power Consumption Monitor Min Avg Max Stock 99.1 327.7 371.6 AI/ML Tuning Recommendations 98.0 351.0 405.4 OpenBenchmarking.org Watts, Fewer Is Better Whisperfile 20Aug24 System Power Consumption Monitor 110 220 330 440 550
Whisperfile CPU Power Consumption Monitor Min Avg Max Stock 3.1 188.6 206.7 AI/ML Tuning Recommendations 52.4 204.4 221.7 OpenBenchmarking.org Watts, Fewer Is Better Whisperfile 20Aug24 CPU Power Consumption Monitor 60 120 180 240 300
Whisperfile System Power Consumption Monitor Min Avg Max Stock 98.2 340.8 387.8 AI/ML Tuning Recommendations 98.4 361.7 409.6 OpenBenchmarking.org Watts, Fewer Is Better Whisperfile 20Aug24 System Power Consumption Monitor 110 220 330 440 550
Whisper.cpp CPU Power Consumption Monitor Min Avg Max Stock 0.8 220.1 235.8 AI/ML Tuning Recommendations 57.9 253.6 273.4 OpenBenchmarking.org Watts, Fewer Is Better Whisper.cpp 1.6.2 CPU Power Consumption Monitor 70 140 210 280 350
Whisper.cpp System Power Consumption Monitor Min Avg Max Stock 98.5 357.9 406.6 AI/ML Tuning Recommendations 98.5 391.4 464.3 OpenBenchmarking.org Watts, Fewer Is Better Whisper.cpp 1.6.2 System Power Consumption Monitor 120 240 360 480 600
Whisper.cpp CPU Power Consumption Monitor Min Avg Max Stock 3.4 244.1 261.6 AI/ML Tuning Recommendations 46.7 283.0 299.0 OpenBenchmarking.org Watts, Fewer Is Better Whisper.cpp 1.6.2 CPU Power Consumption Monitor 80 160 240 320 400
Whisper.cpp System Power Consumption Monitor Min Avg Max Stock 98.2 389.5 469.4 AI/ML Tuning Recommendations 98.7 440.8 504.3 OpenBenchmarking.org Watts, Fewer Is Better Whisper.cpp 1.6.2 System Power Consumption Monitor 130 260 390 520 650
TensorFlow CPU Power Consumption Monitor Min Avg Max Stock 4.1 240.6 255.8 AI/ML Tuning Recommendations 53.0 253.1 264.6 OpenBenchmarking.org Watts, Fewer Is Better TensorFlow 2.16.1 CPU Power Consumption Monitor 70 140 210 280 350
TensorFlow System Power Consumption Monitor Min Avg Max Stock 99.1 442.9 475.7 AI/ML Tuning Recommendations 98.6 460.6 494.4 OpenBenchmarking.org Watts, Fewer Is Better TensorFlow 2.16.1 System Power Consumption Monitor 130 260 390 520 650
TensorFlow CPU Power Consumption Monitor Min Avg Max Stock 1.3 254.8 268.0 AI/ML Tuning Recommendations 53.7 264.7 276.0 OpenBenchmarking.org Watts, Fewer Is Better TensorFlow 2.16.1 CPU Power Consumption Monitor 70 140 210 280 350
TensorFlow System Power Consumption Monitor Min Avg Max Stock 100.5 472.2 518.6 AI/ML Tuning Recommendations 101.6 487.8 535.3 OpenBenchmarking.org Watts, Fewer Is Better TensorFlow 2.16.1 System Power Consumption Monitor 140 280 420 560 700
LiteRT CPU Power Consumption Monitor Min Avg Max Stock 1.1 236.1 263.3 AI/ML Tuning Recommendations 124.4 282.6 304.2 OpenBenchmarking.org Watts, Fewer Is Better LiteRT 2024-10-15 CPU Power Consumption Monitor 80 160 240 320 400
LiteRT System Power Consumption Monitor Min Avg Max Stock 101.7 395.1 407.3 AI/ML Tuning Recommendations 100.3 420.8 461.0 OpenBenchmarking.org Watts, Fewer Is Better LiteRT 2024-10-15 System Power Consumption Monitor 120 240 360 480 600
LiteRT CPU Power Consumption Monitor Min Avg Max Stock 3.7 221.3 238.5 AI/ML Tuning Recommendations 99.2 261.1 279.6 OpenBenchmarking.org Watts, Fewer Is Better LiteRT 2024-10-15 CPU Power Consumption Monitor 70 140 210 280 350
LiteRT System Power Consumption Monitor Min Avg Max Stock 97.1 339.2 364.8 AI/ML Tuning Recommendations 97.7 390.9 414.6 OpenBenchmarking.org Watts, Fewer Is Better LiteRT 2024-10-15 System Power Consumption Monitor 110 220 330 440 550
LiteRT CPU Power Consumption Monitor Min Avg Max Stock 3.6 237.5 264.4 AI/ML Tuning Recommendations 87.0 279.8 303.9 OpenBenchmarking.org Watts, Fewer Is Better LiteRT 2024-10-15 CPU Power Consumption Monitor 80 160 240 320 400
LiteRT System Power Consumption Monitor Min Avg Max Stock 99.2 373.3 408.2 AI/ML Tuning Recommendations 99.6 442.1 460.5 OpenBenchmarking.org Watts, Fewer Is Better LiteRT 2024-10-15 System Power Consumption Monitor 120 240 360 480 600
LiteRT CPU Power Consumption Monitor Min Avg Max Stock 1.4 243.5 272.8 AI/ML Tuning Recommendations 101.0 303.8 326.5 OpenBenchmarking.org Watts, Fewer Is Better LiteRT 2024-10-15 CPU Power Consumption Monitor 80 160 240 320 400
LiteRT System Power Consumption Monitor Min Avg Max Stock 98.4 400.2 424.0 AI/ML Tuning Recommendations 98.1 454.4 495.8 OpenBenchmarking.org Watts, Fewer Is Better LiteRT 2024-10-15 System Power Consumption Monitor 130 260 390 520 650
PyTorch CPU Power Consumption Monitor Min Avg Max Stock 2.9 234.6 290.6 AI/ML Tuning Recommendations 55.1 281.1 335.8 OpenBenchmarking.org Watts, Fewer Is Better PyTorch 2.2.1 CPU Power Consumption Monitor 80 160 240 320 400
PyTorch System Power Consumption Monitor Min Avg Max Stock 98.0 393.2 455.2 AI/ML Tuning Recommendations 98.4 440.0 515.9 OpenBenchmarking.org Watts, Fewer Is Better PyTorch 2.2.1 System Power Consumption Monitor 130 260 390 520 650
PyTorch CPU Power Consumption Monitor Min Avg Max Stock 0.7 264.7 294.6 AI/ML Tuning Recommendations 52.9 311.9 342.9 OpenBenchmarking.org Watts, Fewer Is Better PyTorch 2.2.1 CPU Power Consumption Monitor 80 160 240 320 400
PyTorch System Power Consumption Monitor Min Avg Max Stock 96.8 426.1 462.5 AI/ML Tuning Recommendations 95.7 486.4 526.2 OpenBenchmarking.org Watts, Fewer Is Better PyTorch 2.2.1 System Power Consumption Monitor 130 260 390 520 650
PyTorch CPU Power Consumption Monitor Min Avg Max Stock 1.9 237.5 291.1 AI/ML Tuning Recommendations 53.6 285.5 336.3 OpenBenchmarking.org Watts, Fewer Is Better PyTorch 2.2.1 CPU Power Consumption Monitor 80 160 240 320 400
PyTorch System Power Consumption Monitor Min Avg Max Stock 98.8 395.8 456.8 AI/ML Tuning Recommendations 99.0 441.7 516.7 OpenBenchmarking.org Watts, Fewer Is Better PyTorch 2.2.1 System Power Consumption Monitor 130 260 390 520 650
PyTorch CPU Power Consumption Monitor Min Avg Max Stock 52.1 269.5 294.9 AI/ML Tuning Recommendations 58.3 313.0 344.4 OpenBenchmarking.org Watts, Fewer Is Better PyTorch 2.2.1 CPU Power Consumption Monitor 80 160 240 320 400
PyTorch System Power Consumption Monitor Min Avg Max Stock 98.2 427.3 460.6 AI/ML Tuning Recommendations 98.8 480.2 527.5 OpenBenchmarking.org Watts, Fewer Is Better PyTorch 2.2.1 System Power Consumption Monitor 130 260 390 520 650
oneDNN CPU Power Consumption Monitor Min Avg Max Stock 1.4 147.2 241.7 AI/ML Tuning Recommendations 79.0 177.2 273.9 OpenBenchmarking.org Watts, Fewer Is Better oneDNN 3.6 CPU Power Consumption Monitor 70 140 210 280 350
oneDNN System Power Consumption Monitor Min Avg Max Stock 97.9 288.6 451.9 AI/ML Tuning Recommendations 98.8 244.9 408.8 OpenBenchmarking.org Watts, Fewer Is Better oneDNN 3.6 System Power Consumption Monitor 120 240 360 480 600
oneDNN CPU Power Consumption Monitor Min Avg Max Stock 2.2 174.0 251.2 AI/ML Tuning Recommendations 100.5 202.2 263.1 OpenBenchmarking.org Watts, Fewer Is Better oneDNN 3.6 CPU Power Consumption Monitor 70 140 210 280 350
oneDNN System Power Consumption Monitor Min Avg Max Stock 96.9 310.1 409.7 AI/ML Tuning Recommendations 98.1 331.7 435.9 OpenBenchmarking.org Watts, Fewer Is Better oneDNN 3.6 System Power Consumption Monitor 110 220 330 440 550
oneDNN CPU Power Consumption Monitor Min Avg Max Stock 3.3 107.2 148.5 AI/ML Tuning Recommendations 103.2 116.5 135.9 OpenBenchmarking.org Watts, Fewer Is Better oneDNN 3.6 CPU Power Consumption Monitor 40 80 120 160 200
oneDNN System Power Consumption Monitor Min Avg Max Stock 97.3 220.2 359.1 AI/ML Tuning Recommendations 97.1 186.6 377.1 OpenBenchmarking.org Watts, Fewer Is Better oneDNN 3.6 System Power Consumption Monitor 100 200 300 400 500
oneDNN CPU Power Consumption Monitor Min Avg Max Stock 3.8 146.3 204.5 AI/ML Tuning Recommendations 72.3 169.6 230.8 OpenBenchmarking.org Watts, Fewer Is Better oneDNN 3.6 CPU Power Consumption Monitor 60 120 180 240 300
oneDNN System Power Consumption Monitor Min Avg Max Stock 97.4 255.0 348.1 AI/ML Tuning Recommendations 97.5 268.5 401.0 OpenBenchmarking.org Watts, Fewer Is Better oneDNN 3.6 System Power Consumption Monitor 110 220 330 440 550
oneDNN CPU Power Consumption Monitor Min Avg Max Stock 0.3 155.2 226.2 AI/ML Tuning Recommendations 85.8 188.4 252.5 OpenBenchmarking.org Watts, Fewer Is Better oneDNN 3.6 CPU Power Consumption Monitor 70 140 210 280 350
oneDNN System Power Consumption Monitor Min Avg Max Stock 97.5 274.5 366.1 AI/ML Tuning Recommendations 97.2 303.0 419.6 OpenBenchmarking.org Watts, Fewer Is Better oneDNN 3.6 System Power Consumption Monitor 110 220 330 440 550
oneDNN CPU Power Consumption Monitor Min Avg Max Stock 1.1 203.6 264.7 AI/ML Tuning Recommendations 103.9 226.6 275.0 OpenBenchmarking.org Watts, Fewer Is Better oneDNN 3.6 CPU Power Consumption Monitor 70 140 210 280 350
oneDNN System Power Consumption Monitor Min Avg Max Stock 97.3 337.3 448.7 AI/ML Tuning Recommendations 97.8 365.3 470.9 OpenBenchmarking.org Watts, Fewer Is Better oneDNN 3.6 System Power Consumption Monitor 120 240 360 480 600
oneDNN CPU Power Consumption Monitor Min Avg Max Stock 0.1 197.3 289.9 AI/ML Tuning Recommendations 127.7 222.2 267.0 OpenBenchmarking.org Watts, Fewer Is Better oneDNN 3.6 CPU Power Consumption Monitor 70 140 210 280 350
oneDNN System Power Consumption Monitor Min Avg Max Stock 98.9 329.4 466.9 AI/ML Tuning Recommendations 98.9 361.1 498.3 OpenBenchmarking.org Watts, Fewer Is Better oneDNN 3.6 System Power Consumption Monitor 130 260 390 520 650
Numpy Benchmark CPU Power Consumption Monitor Min Avg Max Stock 3.0 78.8 85.7 AI/ML Tuning Recommendations 47.9 80.7 85.3 OpenBenchmarking.org Watts, Fewer Is Better Numpy Benchmark CPU Power Consumption Monitor 20 40 60 80 100
Numpy Benchmark System Power Consumption Monitor Min Avg Max Stock 97.9 171.2 178.1 AI/ML Tuning Recommendations 98.9 176.8 188.2 OpenBenchmarking.org Watts, Fewer Is Better Numpy Benchmark System Power Consumption Monitor 50 100 150 200 250
ONNX Runtime CPU Power Consumption Monitor Min Avg Max Stock 1.5 203.6 221.2 AI/ML Tuning Recommendations 82.2 237.3 256.8 OpenBenchmarking.org Watts, Fewer Is Better ONNX Runtime 1.19 CPU Power Consumption Monitor 70 140 210 280 350
ONNX Runtime System Power Consumption Monitor Min Avg Max Stock 98.4 328.4 350.0 AI/ML Tuning Recommendations 99.3 375.5 395.9 OpenBenchmarking.org Watts, Fewer Is Better ONNX Runtime 1.19 System Power Consumption Monitor 110 220 330 440 550
ONNX Runtime CPU Power Consumption Monitor Min Avg Max Stock 3.0 248.9 279.9 AI/ML Tuning Recommendations 77.0 270.8 300.9 OpenBenchmarking.org Watts, Fewer Is Better ONNX Runtime 1.19 CPU Power Consumption Monitor 80 160 240 320 400
ONNX Runtime System Power Consumption Monitor Min Avg Max Stock 98.9 429.9 477.0 AI/ML Tuning Recommendations 98.5 460.1 509.9 OpenBenchmarking.org Watts, Fewer Is Better ONNX Runtime 1.19 System Power Consumption Monitor 130 260 390 520 650
XNNPACK CPU Power Consumption Monitor Min Avg Max Stock 1.9 241.3 263.5 AI/ML Tuning Recommendations 0.0 279.8 305.1 OpenBenchmarking.org Watts, Fewer Is Better XNNPACK b7b048 CPU Power Consumption Monitor 80 160 240 320 400
XNNPACK System Power Consumption Monitor Min Avg Max Stock 99.6 377.9 403.8 AI/ML Tuning Recommendations 100.4 424.2 458.8 OpenBenchmarking.org Watts, Fewer Is Better XNNPACK b7b048 System Power Consumption Monitor 120 240 360 480 600
OpenVINO GenAI Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU - Time To First Token OpenBenchmarking.org ms, Fewer Is Better OpenVINO GenAI 2024.5 Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU - Time To First Token Stock AI/ML Tuning Recommendations 6 12 18 24 30 SE +/- 0.14, N = 4 SE +/- 0.07, N = 4 24.17 25.66
OpenVINO GenAI Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU - Time Per Output Token OpenBenchmarking.org ms, Fewer Is Better OpenVINO GenAI 2024.5 Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU - Time Per Output Token Stock AI/ML Tuning Recommendations 4 8 12 16 20 SE +/- 0.05, N = 4 SE +/- 0.04, N = 4 17.98 17.71
OpenVINO GenAI Model: Falcon-7b-instruct-int4-ov - Device: CPU - Time To First Token OpenBenchmarking.org ms, Fewer Is Better OpenVINO GenAI 2024.5 Model: Falcon-7b-instruct-int4-ov - Device: CPU - Time To First Token Stock AI/ML Tuning Recommendations 7 14 21 28 35 SE +/- 0.03, N = 3 SE +/- 0.18, N = 3 29.07 30.70
OpenVINO GenAI Model: Falcon-7b-instruct-int4-ov - Device: CPU - Time Per Output Token OpenBenchmarking.org ms, Fewer Is Better OpenVINO GenAI 2024.5 Model: Falcon-7b-instruct-int4-ov - Device: CPU - Time Per Output Token Stock AI/ML Tuning Recommendations 5 10 15 20 25 SE +/- 0.04, N = 3 SE +/- 0.07, N = 3 19.60 19.56
OpenVINO GenAI Model: Gemma-7b-int4-ov - Device: CPU - Time To First Token OpenBenchmarking.org ms, Fewer Is Better OpenVINO GenAI 2024.5 Model: Gemma-7b-int4-ov - Device: CPU - Time To First Token Stock AI/ML Tuning Recommendations 8 16 24 32 40 SE +/- 0.20, N = 3 SE +/- 0.08, N = 3 36.13 35.43
OpenVINO GenAI Model: Gemma-7b-int4-ov - Device: CPU - Time Per Output Token OpenBenchmarking.org ms, Fewer Is Better OpenVINO GenAI 2024.5 Model: Gemma-7b-int4-ov - Device: CPU - Time Per Output Token Stock AI/ML Tuning Recommendations 6 12 18 24 30 SE +/- 0.09, N = 3 SE +/- 0.06, N = 3 26.43 26.31
ONNX Runtime Model: ResNet50 v1-12-int8 - Device: CPU - Executor: Standard OpenBenchmarking.org Inference Time Cost (ms), Fewer Is Better ONNX Runtime 1.19 Model: ResNet50 v1-12-int8 - Device: CPU - Executor: Standard Stock AI/ML Tuning Recommendations 0.8153 1.6306 2.4459 3.2612 4.0765 SE +/- 0.03395, N = 7 SE +/- 0.01794, N = 3 3.62341 3.50697 1. (CXX) g++ options: -O3 -march=native -ffunction-sections -fdata-sections -mtune=native -flto=auto -fno-fat-lto-objects -ldl -lrt
ONNX Runtime Model: ResNet101_DUC_HDC-12 - Device: CPU - Executor: Standard OpenBenchmarking.org Inference Time Cost (ms), Fewer Is Better ONNX Runtime 1.19 Model: ResNet101_DUC_HDC-12 - Device: CPU - Executor: Standard Stock AI/ML Tuning Recommendations 40 80 120 160 200 SE +/- 3.66, N = 15 SE +/- 3.81, N = 15 165.09 158.53 1. (CXX) g++ options: -O3 -march=native -ffunction-sections -fdata-sections -mtune=native -flto=auto -fno-fat-lto-objects -ldl -lrt
Phoronix Test Suite v10.8.5