Xeon Max AMX HBM2e

Benchmarks for a future article on Phoronix.

Compare your own system(s) to this result file with the Phoronix Test Suite by running the command: phoronix-test-suite benchmark 2306302-NE-XEONMAXAM96
Jump To Table - Results

View

Do Not Show Noisy Results
Do Not Show Results With Incomplete Data
Do Not Show Results With Little Change/Spread
List Notable Results
Show Result Confidence Charts

Limit displaying results to tests within:

CPU Massive 2 Tests
Creator Workloads 2 Tests
HPC - High Performance Computing 4 Tests
Machine Learning 4 Tests
Multi-Core 2 Tests
Intel oneAPI 2 Tests
Python Tests 3 Tests

Statistics

Show Overall Harmonic Mean(s)
Show Overall Geometric Mean
Show Geometric Means Per-Suite/Category
Show Wins / Losses Counts (Pie Chart)
Normalize Results
Remove Outliers Before Calculating Averages

Graph Settings

Force Line Graphs Where Applicable
Convert To Scalar Where Applicable
Disable Color Branding
Prefer Vertical Bar Graphs
No Box Plots
On Line Graphs With Missing Data, Connect The Line Gaps

Additional Graphs

Show Perf Per Core/Thread Calculation Graphs Where Applicable
Show Perf Per Clock Calculation Graphs Where Applicable

Multi-Way Comparison

Condense Multi-Option Tests Into Single Result Graphs
Condense Test Profiles With Multiple Version Results Into Single Result Graphs

Table

Show Detailed System Result Table

Run Management

Highlight
Result
Hide
Result
Result
Identifier
Performance Per
Dollar
Date
Run
  Test
  Duration
EPYC 9654 2P
June 28 2023
  6 Hours, 59 Minutes
EPYC 9554 2P
June 28 2023
  5 Hours, 42 Minutes
Xeon Max 9480 2P, HBM Caching
June 29 2023
  13 Hours, 47 Minutes
Invert Hiding All Results Option
  8 Hours, 49 Minutes

Only show results where is faster than
Only show results matching title/arguments (delimit multiple options with a comma):
Do not show results matching title/arguments (delimit multiple options with a comma):


Xeon Max AMX HBM2eProcessorMotherboardChipsetMemoryDiskGraphicsMonitorNetworkOSKernelDesktopDisplay ServerCompilerFile-SystemScreen ResolutionEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching2 x AMD EPYC 9654 96-Core @ 2.40GHz (192 Cores / 384 Threads)AMD Titanite_4G (RTI1007B BIOS)AMD Device 14a41520GB7682GB INTEL SSDPF2KX076TZASPEEDVGA HDMIBroadcom NetXtreme BCM5720 PCIeUbuntu 23.046.2.0-20-generic (x86_64)GNOME Shell 44.0X Server 1.21.1.7GCC 12.2.0ext41920x10802 x AMD EPYC 9554 64-Core @ 3.10GHz (128 Cores / 256 Threads)2 x Intel Xeon Max 9480 @ 3.50GHz (112 Cores / 224 Threads)Supermicro X13DEM v1.10 (1.3 BIOS)Intel Device 1bce512GBVE2282 x Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200GbOpenBenchmarking.orgKernel Details- Transparent Huge Pages: madviseCompiler Details- --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details- EPYC 9654 2P: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa101121- EPYC 9554 2P: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa101121- Xeon Max 9480 2P, HBM Caching: Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c0001d1Python Details- Python 3.11.2Security Details- EPYC 9654 2P: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected - EPYC 9554 2P: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected - Xeon Max 9480 2P, HBM Caching: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

EPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM CachingResult OverviewPhoronix Test Suite100%171%242%313%384%oneDNNlibxsmmTensorFlowONNX RuntimeOpenVINO

EPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM CachingPer Watt Result OverviewPhoronix Test Suite100%168%236%303%TensorFlowTensorFlowTensorFlowTensorFlowTensorFlowTensorFlowTensorFlowTensorFlowlibxsmmCPU - 512 - ResNet-50CPU - 512 - GoogLeNetCPU - 16 - ResNet-50CPU - 16 - GoogLeNetCPU - 512 - AlexNetCPU - 512 - VGG-16CPU - 16 - AlexNetCPU - 16 - VGG-16128

Xeon Max AMX HBM2eonednn: Deconvolution Batch shapes_3d - bf16bf16bf16 - CPUopenvino: Vehicle Detection FP16-INT8 - CPUonednn: IP Shapes 3D - bf16bf16bf16 - CPUopenvino: Person Vehicle Bike Detection FP16 - CPUtensorflow: CPU - 512 - VGG-16openvino: Vehicle Detection FP16 - CPUonednn: Recurrent Neural Network Inference - bf16bf16bf16 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUtensorflow: CPU - 512 - AlexNettensorflow: CPU - 512 - GoogLeNettensorflow: CPU - 512 - ResNet-50libxsmm: 128openvino: Vehicle Detection FP16 - CPUopenvino: Face Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16 - CPUopenvino: Face Detection FP16 - CPUonednn: Convolution Batch Shapes Auto - bf16bf16bf16 - CPUopenvino: Vehicle Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16 - CPUopenvino: Face Detection FP16-INT8 - CPUopenvino: Machine Translation EN To DE FP16 - CPUtensorflow: CPU - 16 - ResNet-50openvino: Age Gender Recognition Retail 0013 FP16 - CPUopenvino: Face Detection FP16 - CPUopenvino: Person Vehicle Bike Detection FP16 - CPUopenvino: Person Detection FP32 - CPUopenvino: Person Detection FP16 - CPUopenvino: Person Detection FP16 - CPUopenvino: Person Detection FP32 - CPUopenvino: Age Gender Recognition Retail 0013 FP16 - CPUonnx: ArcFace ResNet-100 - CPU - Standardopenvino: Machine Translation EN To DE FP16 - CPUlibxsmm: 64libxsmm: 32libxsmm: 256onnx: ResNet50 v1-12-int8 - CPU - Standardonnx: ResNet50 v1-12-int8 - CPU - Standardonnx: ArcFace ResNet-100 - CPU - Standardonnx: fcn-resnet101-11 - CPU - Standardonnx: fcn-resnet101-11 - CPU - Standardonnx: CaffeNet 12-int8 - CPU - Standardonnx: CaffeNet 12-int8 - CPU - Standardonnx: bertsquad-12 - CPU - Standardonnx: bertsquad-12 - CPU - Standardonnx: yolov4 - CPU - Standardonnx: yolov4 - CPU - Standardonnx: GPT-2 - CPU - Standardonnx: GPT-2 - CPU - Standardopenvino: Age Gender Recognition Retail 0013 FP16-INT8 - CPUopenvino: Age Gender Recognition Retail 0013 FP16-INT8 - CPUtensorflow: CPU - 16 - GoogLeNettensorflow: CPU - 16 - AlexNettensorflow: CPU - 16 - VGG-16onednn: Recurrent Neural Network Training - bf16bf16bf16 - CPUonednn: Deconvolution Batch shapes_1d - bf16bf16bf16 - CPUonednn: IP Shapes 1D - bf16bf16bf16 - CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching0.7321004.363.802095.25120.627390.972790.909.951757.16535.52171.563426.16.49191.419843.03471.820.46105310985.1719170.614.86250.39950.9425.25146867.48101.569133.221098.961092.9743.6443.400.5519.308450.422270.21298.04379.66.87053146.55551.7898212.8634.697972.30611435.19498.995710.17944204.0914.900339.68904103.1840.99124618.3264.49166.2047.272492.692.5332616.98230.4572723.542.205964.29109.895981.57925.7678.091813.68593.20188.174112.95.34155.288039.57389.530.3113789010.4515659.323.97205.71786.0133.40162207.9782.047447.60825.52832.0238.2438.530.5224.012240.672692.71402.24603.36.55989152.43941.6445214.6064.659822.01949495.24085.900911.6871195.8995.105057.64735132.7210.88128593.2494.41257.9460.311305.661.7616211.196693.5352520.2391.981417.8330.192315.121325.453.47718.54236.4476.931814.312.07334.8916941.99232.013.673565528.0330938.506.44333.51590.9240.15108515.12120.406265.77819.76834.9733.4033.970.4423.772947.298.02166125.28442.1801191.6235.335141.78296562.21887.763811.4576130.3707.700464.70849218.6921.4059457.1498.43230.1121.0915294.60.4830757.88016OpenBenchmarking.org

oneDNN

This is a test of the Intel oneDNN as an Intel-optimized library for Deep Neural Networks and making use of its built-in benchdnn functionality. The result is the total perf time reported. Intel oneDNN was formerly known as DNNL (Deep Neural Network Library) and MKL-DNN before being rebranded as part of the Intel oneAPI toolkit. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgms, Fewer Is BetteroneDNN 3.1Harness: Deconvolution Batch shapes_3d - Data Type: bf16bf16bf16 - Engine: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching0.79541.59082.38623.18163.977SE +/- 0.004782, N = 9SE +/- 0.002054, N = 9SE +/- 0.029518, N = 150.7321000.4572723.535250MIN: 2.461. (CXX) g++ options: -O3 -march=native -fopenmp -msse4.1 -fPIC -pie -ldl -lpthread

OpenVINO

This is a test of the Intel OpenVINO, a toolkit around neural networks, using its built-in benchmarking support and analyzing the throughput and latency for various models. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Vehicle Detection FP16-INT8 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching510152025SE +/- 0.00, N = 3SE +/- 0.00, N = 3SE +/- 0.09, N = 34.363.5420.23MIN: 3.52 / MAX: 38.49MIN: 3.45 / MAX: 21.7MIN: 12.69 / MAX: 166.761. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

oneDNN

This is a test of the Intel oneDNN as an Intel-optimized library for Deep Neural Networks and making use of its built-in benchdnn functionality. The result is the total perf time reported. Intel oneDNN was formerly known as DNNL (Deep Neural Network Library) and MKL-DNN before being rebranded as part of the Intel oneAPI toolkit. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgms, Fewer Is BetteroneDNN 3.1Harness: IP Shapes 3D - Data Type: bf16bf16bf16 - Engine: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching20406080100SE +/- 0.03459, N = 15SE +/- 0.02374, N = 15SE +/- 0.61972, N = 53.802092.2059691.98140MIN: 79.161. (CXX) g++ options: -O3 -march=native -fopenmp -msse4.1 -fPIC -pie -ldl -lpthread

OpenVINO

This is a test of the Intel OpenVINO, a toolkit around neural networks, using its built-in benchmarking support and analyzing the throughput and latency for various models. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Person Vehicle Bike Detection FP16 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching48121620SE +/- 0.00, N = 3SE +/- 0.01, N = 3SE +/- 0.02, N = 35.254.2917.83MIN: 4.36 / MAX: 23.21MIN: 4.11 / MAX: 21.83MIN: 13.06 / MAX: 147.361. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

TensorFlow

This is a benchmark of the TensorFlow deep learning framework using the TensorFlow reference benchmarks (tensorflow/benchmarks with tf_cnn_benchmarks.py). Note with the Phoronix Test Suite there is also pts/tensorflow-lite for benchmarking the TensorFlow Lite binaries if desired for complementary metrics. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgimages/sec, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 512 - Model: VGG-16EPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching306090120150SE +/- 0.08, N = 3SE +/- 1.46, N = 3SE +/- 0.33, N = 5120.62109.8930.19

OpenVINO

This is a test of the Intel OpenVINO, a toolkit around neural networks, using its built-in benchmarking support and analyzing the throughput and latency for various models. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Vehicle Detection FP16 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching16003200480064008000SE +/- 2.12, N = 3SE +/- 3.49, N = 3SE +/- 17.29, N = 137390.975981.572315.121. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

oneDNN

This is a test of the Intel oneDNN as an Intel-optimized library for Deep Neural Networks and making use of its built-in benchdnn functionality. The result is the total perf time reported. Intel oneDNN was formerly known as DNNL (Deep Neural Network Library) and MKL-DNN before being rebranded as part of the Intel oneAPI toolkit. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgms, Fewer Is BetteroneDNN 3.1Harness: Recurrent Neural Network Inference - Data Type: bf16bf16bf16 - Engine: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching6001200180024003000SE +/- 29.80, N = 15SE +/- 2.71, N = 3SE +/- 20.64, N = 122790.90925.771325.45MIN: 2391.98MIN: 909.67MIN: 1025.181. (CXX) g++ options: -O3 -march=native -fopenmp -msse4.1 -fPIC -pie -ldl -lpthread

OpenVINO

This is a test of the Intel OpenVINO, a toolkit around neural networks, using its built-in benchmarking support and analyzing the throughput and latency for various models. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching3691215SE +/- 0.00, N = 3SE +/- 0.00, N = 3SE +/- 0.00, N = 39.958.093.47MIN: 8.36 / MAX: 40.39MIN: 7.89 / MAX: 45.39MIN: 2.48 / MAX: 56.571. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

TensorFlow

This is a benchmark of the TensorFlow deep learning framework using the TensorFlow reference benchmarks (tensorflow/benchmarks with tf_cnn_benchmarks.py). Note with the Phoronix Test Suite there is also pts/tensorflow-lite for benchmarking the TensorFlow Lite binaries if desired for complementary metrics. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgimages/sec, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 512 - Model: AlexNetEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching400800120016002000SE +/- 16.13, N = 7SE +/- 12.94, N = 3SE +/- 9.04, N = 31757.161813.68718.54

OpenBenchmarking.orgimages/sec, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 512 - Model: GoogLeNetEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching130260390520650SE +/- 0.69, N = 3SE +/- 0.96, N = 3SE +/- 1.79, N = 3535.52593.20236.44

OpenBenchmarking.orgimages/sec, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 512 - Model: ResNet-50EPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching4080120160200SE +/- 0.26, N = 3SE +/- 0.70, N = 3SE +/- 0.66, N = 3171.56188.1776.93

libxsmm

Libxsmm is an open-source library for specialized dense and sparse matrix operations and deep learning primitives. Libxsmm supports making use of Intel AMX, AVX-512, and other modern CPU instruction set capabilities. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgGFLOPS/s, More Is Betterlibxsmm 2-1.17-3645M N K: 128EPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching9001800270036004500SE +/- 29.11, N = 3SE +/- 21.03, N = 3SE +/- 22.78, N = 93426.14112.91814.31. (CXX) g++ options: -dynamic -Bstatic -static-libgcc -lgomp -lm -lrt -ldl -lquadmath -lstdc++ -pthread -fPIC -std=c++14 -O2 -fopenmp-simd -funroll-loops -ftree-vectorize -fdata-sections -ffunction-sections -fvisibility=hidden -msse4.2

OpenVINO

This is a test of the Intel OpenVINO, a toolkit around neural networks, using its built-in benchmarking support and analyzing the throughput and latency for various models. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Vehicle Detection FP16 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching3691215SE +/- 0.00, N = 3SE +/- 0.00, N = 3SE +/- 0.08, N = 136.495.3412.07MIN: 5.03 / MAX: 55.65MIN: 4.89 / MAX: 34.7MIN: 7.38 / MAX: 250.921. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching70140210280350SE +/- 0.06, N = 3SE +/- 0.14, N = 3SE +/- 0.45, N = 3191.41155.28334.891. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching4K8K12K16K20KSE +/- 2.32, N = 3SE +/- 1.22, N = 3SE +/- 13.63, N = 39843.038039.5716941.991. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Face Detection FP16 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching100200300400500SE +/- 0.93, N = 3SE +/- 0.19, N = 3SE +/- 0.80, N = 3471.82389.53232.01MIN: 431.51 / MAX: 569.3MIN: 380.82 / MAX: 432.57MIN: 138.61 / MAX: 770.451. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

oneDNN

This is a test of the Intel oneDNN as an Intel-optimized library for Deep Neural Networks and making use of its built-in benchdnn functionality. The result is the total perf time reported. Intel oneDNN was formerly known as DNNL (Deep Neural Network Library) and MKL-DNN before being rebranded as part of the Intel oneAPI toolkit. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgms, Fewer Is BetteroneDNN 3.1Harness: Convolution Batch Shapes Auto - Data Type: bf16bf16bf16 - Engine: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching0.82661.65322.47983.30644.133SE +/- 0.003345, N = 15SE +/- 0.002531, N = 7SE +/- 0.025287, N = 70.4610530.3113783.673560MIN: 3.031. (CXX) g++ options: -O3 -march=native -fopenmp -msse4.1 -fPIC -pie -ldl -lpthread

OpenVINO

This is a test of the Intel OpenVINO, a toolkit around neural networks, using its built-in benchmarking support and analyzing the throughput and latency for various models. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Vehicle Detection FP16-INT8 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching2K4K6K8K10KSE +/- 0.53, N = 3SE +/- 2.20, N = 3SE +/- 25.18, N = 310985.179010.455528.031. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching7K14K21K28K35KSE +/- 4.81, N = 3SE +/- 3.60, N = 3SE +/- 38.32, N = 319170.6115659.3230938.501. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching246810SE +/- 0.00, N = 3SE +/- 0.00, N = 3SE +/- 0.01, N = 34.863.976.44MIN: 4.02 / MAX: 36.01MIN: 3.87 / MAX: 17.35MIN: 4.24 / MAX: 49.431. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching70140210280350SE +/- 0.04, N = 3SE +/- 0.22, N = 3SE +/- 0.44, N = 3250.39205.71333.51MIN: 222.36 / MAX: 303.72MIN: 202.38 / MAX: 252.14MIN: 285.32 / MAX: 516.281. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Machine Translation EN To DE FP16 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching2004006008001000SE +/- 0.77, N = 3SE +/- 1.56, N = 3SE +/- 0.19, N = 3950.94786.01590.921. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

TensorFlow

This is a benchmark of the TensorFlow deep learning framework using the TensorFlow reference benchmarks (tensorflow/benchmarks with tf_cnn_benchmarks.py). Note with the Phoronix Test Suite there is also pts/tensorflow-lite for benchmarking the TensorFlow Lite binaries if desired for complementary metrics. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgimages/sec, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 16 - Model: ResNet-50EPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching918273645SE +/- 0.31, N = 3SE +/- 0.23, N = 3SE +/- 0.32, N = 325.2533.4040.15

OpenVINO

This is a test of the Intel OpenVINO, a toolkit around neural networks, using its built-in benchmarking support and analyzing the throughput and latency for various models. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Age Gender Recognition Retail 0013 FP16 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching30K60K90K120K150KSE +/- 1825.86, N = 3SE +/- 685.52, N = 3SE +/- 967.90, N = 3146867.48162207.97108515.121. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Face Detection FP16 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching306090120150SE +/- 0.25, N = 3SE +/- 0.04, N = 3SE +/- 0.41, N = 3101.5682.04120.401. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Person Vehicle Bike Detection FP16 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching2K4K6K8K10KSE +/- 3.38, N = 3SE +/- 8.54, N = 3SE +/- 8.17, N = 39133.227447.606265.771. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Person Detection FP32 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching2004006008001000SE +/- 5.33, N = 3SE +/- 1.07, N = 3SE +/- 7.20, N = 31098.96825.52819.76MIN: 825.62 / MAX: 1835.76MIN: 747.05 / MAX: 1238.39MIN: 516.88 / MAX: 2598.391. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Person Detection FP16 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching2004006008001000SE +/- 5.11, N = 3SE +/- 5.30, N = 3SE +/- 6.17, N = 31092.97832.02834.97MIN: 825.28 / MAX: 1814.82MIN: 743.95 / MAX: 1258.17MIN: 558.93 / MAX: 2258.391. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Person Detection FP16 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching1020304050SE +/- 0.21, N = 3SE +/- 0.23, N = 3SE +/- 0.24, N = 343.6438.2433.401. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Person Detection FP32 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching1020304050SE +/- 0.20, N = 3SE +/- 0.04, N = 3SE +/- 0.31, N = 343.4038.5333.971. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Age Gender Recognition Retail 0013 FP16 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching0.12380.24760.37140.49520.619SE +/- 0.00, N = 3SE +/- 0.00, N = 3SE +/- 0.00, N = 30.550.520.44MIN: 0.5 / MAX: 45.73MIN: 0.49 / MAX: 50.12MIN: 0.33 / MAX: 42.851. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

ONNX Runtime

ONNX Runtime is developed by Microsoft and partners as a open-source, cross-platform, high performance machine learning inferencing and training accelerator. This test profile runs the ONNX Runtime with various models available from the ONNX Model Zoo. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgInferences Per Second, More Is BetterONNX Runtime 1.14Model: ArcFace ResNet-100 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching612182430SE +/- 0.09, N = 3SE +/- 0.09, N = 3SE +/- 0.34, N = 1519.3124.0123.771. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

OpenVINO

This is a test of the Intel OpenVINO, a toolkit around neural networks, using its built-in benchmarking support and analyzing the throughput and latency for various models. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Machine Translation EN To DE FP16 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching1122334455SE +/- 0.04, N = 3SE +/- 0.08, N = 3SE +/- 0.01, N = 350.4240.6747.29MIN: 39.07 / MAX: 280.94MIN: 34.96 / MAX: 124.31MIN: 32.14 / MAX: 577.781. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

libxsmm

Libxsmm is an open-source library for specialized dense and sparse matrix operations and deep learning primitives. Libxsmm supports making use of Intel AMX, AVX-512, and other modern CPU instruction set capabilities. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgGFLOPS/s, More Is Betterlibxsmm 2-1.17-3645M N K: 64EPYC 9654 2PEPYC 9554 2P6001200180024003000SE +/- 33.30, N = 15SE +/- 5.63, N = 72270.22692.71. (CXX) g++ options: -dynamic -Bstatic -static-libgcc -lgomp -lm -lrt -ldl -lquadmath -lstdc++ -pthread -fPIC -std=c++14 -O2 -fopenmp-simd -funroll-loops -ftree-vectorize -fdata-sections -ffunction-sections -fvisibility=hidden -msse4.2

OpenBenchmarking.orgGFLOPS/s, More Is Betterlibxsmm 2-1.17-3645M N K: 32EPYC 9654 2PEPYC 9554 2P30060090012001500SE +/- 9.98, N = 15SE +/- 4.06, N = 81298.01402.21. (CXX) g++ options: -dynamic -Bstatic -static-libgcc -lgomp -lm -lrt -ldl -lquadmath -lstdc++ -pthread -fPIC -std=c++14 -O2 -fopenmp-simd -funroll-loops -ftree-vectorize -fdata-sections -ffunction-sections -fvisibility=hidden -msse4.2

OpenBenchmarking.orgGFLOPS/s, More Is Betterlibxsmm 2-1.17-3645M N K: 256EPYC 9654 2PEPYC 9554 2P10002000300040005000SE +/- 47.44, N = 5SE +/- 34.70, N = 34379.64603.31. (CXX) g++ options: -dynamic -Bstatic -static-libgcc -lgomp -lm -lrt -ldl -lquadmath -lstdc++ -pthread -fPIC -std=c++14 -O2 -fopenmp-simd -funroll-loops -ftree-vectorize -fdata-sections -ffunction-sections -fvisibility=hidden -msse4.2

System Power Consumption Monitor

OpenBenchmarking.orgWattsSystem Power Consumption MonitorPhoronix Test Suite System MonitoringXeon Max 9480 2P, HBM Caching160320480640800Min: 260 / Avg: 751.74 / Max: 907

CPU Peak Freq (Highest CPU Core Frequency) Monitor

OpenBenchmarking.orgMegahertzCPU Peak Freq (Highest CPU Core Frequency) MonitorPhoronix Test Suite System MonitoringXeon Max 9480 2P, HBM Caching6001200180024003000Min: 1100 / Avg: 3207.85 / Max: 3577

ONNX Runtime

OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14System Power Consumption MonitorXeon Max 9480 2P, HBM Caching150300450600750Min: 410 / Avg: 784.18 / Max: 830

OpenBenchmarking.orgMegahertz, More Is BetterONNX Runtime 1.14CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, HBM Caching6001200180024003000Min: 2006 / Avg: 3497.11 / Max: 3509

OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14System Power Consumption MonitorXeon Max 9480 2P, HBM Caching150300450600750Min: 400 / Avg: 778.48 / Max: 853

OpenBenchmarking.orgMegahertz, More Is BetterONNX Runtime 1.14CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, HBM Caching6001200180024003000Min: 1900 / Avg: 3482.84 / Max: 3511

OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14System Power Consumption MonitorXeon Max 9480 2P, HBM Caching150300450600750Min: 410 / Avg: 784.54 / Max: 844

OpenBenchmarking.orgMegahertz, More Is BetterONNX Runtime 1.14CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, HBM Caching6001200180024003000Min: 3500 / Avg: 3500.01 / Max: 3507

OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14System Power Consumption MonitorXeon Max 9480 2P, HBM Caching150300450600750Min: 399 / Avg: 783.38 / Max: 828

OpenBenchmarking.orgMegahertz, More Is BetterONNX Runtime 1.14CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, HBM Caching6001200180024003000Min: 3111 / Avg: 3499.75 / Max: 3508

MinAvgMaxXeon Max 9480 2P, HBM Caching390783827OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14System Power Consumption Monitor2004006008001000

MinAvgMaxXeon Max 9480 2P, HBM Caching350035003507OpenBenchmarking.orgMegahertz, More Is BetterONNX Runtime 1.14CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14System Power Consumption MonitorXeon Max 9480 2P, HBM Caching140280420560700Min: 281 / Avg: 720.02 / Max: 770

OpenBenchmarking.orgMegahertz, More Is BetterONNX Runtime 1.14CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, HBM Caching6001200180024003000Min: 3500 / Avg: 3500.97 / Max: 3517

OpenVINO

MinAvgMaxXeon Max 9480 2P, HBM Caching268742856OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor2004006008001000

MinAvgMaxXeon Max 9480 2P, HBM Caching110020993500OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

TensorFlow

MinAvgMaxXeon Max 9480 2P, HBM Caching405693801OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12System Power Consumption Monitor2004006008001000

MinAvgMaxXeon Max 9480 2P, HBM Caching266634893505OpenBenchmarking.orgMegahertz, More Is BetterTensorFlow 2.12CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

MinAvgMaxXeon Max 9480 2P, HBM Caching408636785OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12System Power Consumption Monitor2004006008001000

MinAvgMaxXeon Max 9480 2P, HBM Caching260033063511OpenBenchmarking.orgMegahertz, More Is BetterTensorFlow 2.12CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12System Power Consumption MonitorXeon Max 9480 2P, HBM Caching140280420560700Min: 410 / Avg: 755.75 / Max: 822

OpenBenchmarking.orgMegahertz, More Is BetterTensorFlow 2.12CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, HBM Caching6001200180024003000Min: 2600 / Avg: 3476.25 / Max: 3555

oneDNN

OpenBenchmarking.orgWatts, Fewer Is BetteroneDNN 3.1System Power Consumption MonitorXeon Max 9480 2P, HBM Caching160320480640800Min: 410 / Avg: 782.09 / Max: 889

OpenBenchmarking.orgMegahertz, More Is BetteroneDNN 3.1CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, HBM Caching6001200180024003000Min: 1951 / Avg: 2982.06 / Max: 3514

MinAvgMaxXeon Max 9480 2P, HBM Caching404727850OpenBenchmarking.orgWatts, Fewer Is BetteroneDNN 3.1System Power Consumption Monitor2004006008001000

MinAvgMaxXeon Max 9480 2P, HBM Caching216626513508OpenBenchmarking.orgMegahertz, More Is BetteroneDNN 3.1CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

MinAvgMaxXeon Max 9480 2P, HBM Caching385721838OpenBenchmarking.orgWatts, Fewer Is BetteroneDNN 3.1System Power Consumption Monitor2004006008001000

MinAvgMaxXeon Max 9480 2P, HBM Caching260027973507OpenBenchmarking.orgMegahertz, More Is BetteroneDNN 3.1CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

CPU Power Consumption Monitor

OpenBenchmarking.orgWattsCPU Power Consumption MonitorPhoronix Test Suite System MonitoringEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching160320480640800Min: 31.88 / Avg: 448.81 / Max: 725.51Min: 34.39 / Avg: 413.82 / Max: 710.63Min: 87.33 / Avg: 548.37 / Max: 899.87

ONNX Runtime

OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14CPU Power Consumption MonitorEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching110220330440550Min: 31.88 / Avg: 465.24 / Max: 489.61Min: 47.64 / Avg: 368.58 / Max: 386.5Min: 270.52 / Avg: 569.27 / Max: 613.29

OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: ResNet50 v1-12-int8 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching246810SE +/- 0.15229, N = 15SE +/- 0.04787, N = 3SE +/- 0.15815, N = 156.870536.559898.021661. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

OpenBenchmarking.orgInferences Per Second, More Is BetterONNX Runtime 1.14Model: ResNet50 v1-12-int8 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching306090120150SE +/- 3.31, N = 15SE +/- 1.11, N = 3SE +/- 2.29, N = 15146.56152.44125.281. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

MinAvgMaxEPYC 9654 2P57514546EPYC 9554 2P47420447Xeon Max 9480 2P, HBM Caching244569665OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14CPU Power Consumption Monitor2004006008001000

OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: fcn-resnet101-11 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching50100150200250SE +/- 1.01, N = 3SE +/- 1.01, N = 3SE +/- 8.54, N = 15212.86214.61191.621. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

OpenBenchmarking.orgInferences Per Second, More Is BetterONNX Runtime 1.14Model: fcn-resnet101-11 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching1.20042.40083.60124.80166.002SE +/- 0.02211, N = 3SE +/- 0.02211, N = 3SE +/- 0.18914, N = 154.697974.659825.335141. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14CPU Power Consumption MonitorEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching110220330440550Min: 56.74 / Avg: 454.2 / Max: 477.67Min: 47.45 / Avg: 357.73 / Max: 374.81Min: 261.61 / Avg: 573.75 / Max: 607.12

OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: CaffeNet 12-int8 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching0.51891.03781.55672.07562.5945SE +/- 0.03734, N = 15SE +/- 0.02472, N = 4SE +/- 0.02850, N = 152.306112.019491.782961. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

OpenBenchmarking.orgInferences Per Second, More Is BetterONNX Runtime 1.14Model: CaffeNet 12-int8 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching120240360480600SE +/- 7.41, N = 15SE +/- 5.94, N = 4SE +/- 8.12, N = 15435.19495.24562.221. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14CPU Power Consumption MonitorEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching110220330440550Min: 57.25 / Avg: 394.67 / Max: 418.06Min: 47.62 / Avg: 368.65 / Max: 387.58Min: 264.43 / Avg: 572.92 / Max: 616.32

OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: bertsquad-12 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching20406080100SE +/- 2.35, N = 15SE +/- 1.46, N = 15SE +/- 1.73, N = 1599.0085.9087.761. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

OpenBenchmarking.orgInferences Per Second, More Is BetterONNX Runtime 1.14Model: bertsquad-12 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching3691215SE +/- 0.24, N = 15SE +/- 0.19, N = 15SE +/- 0.23, N = 1510.1811.6911.461. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

MinAvgMaxEPYC 9654 2P57456482EPYC 9554 2P47360380Xeon Max 9480 2P, HBM Caching262573626OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14CPU Power Consumption Monitor2004006008001000

OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: yolov4 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching4080120160200SE +/- 1.68, N = 3SE +/- 1.35, N = 3SE +/- 2.64, N = 12204.09195.90130.371. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

OpenBenchmarking.orgInferences Per Second, More Is BetterONNX Runtime 1.14Model: yolov4 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching246810SE +/- 0.04001, N = 3SE +/- 0.03523, N = 3SE +/- 0.13594, N = 124.900335.105057.700461. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

MinAvgMaxEPYC 9654 2P58.7257.5270.6EPYC 9554 2P46.8222.0233.4Xeon Max 9480 2P, HBM Caching169.4521.4555.8OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14CPU Power Consumption Monitor140280420560700

OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: GPT-2 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching3691215SE +/- 0.00311, N = 3SE +/- 0.25707, N = 15SE +/- 0.23765, N = 159.689047.647354.708491. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

OpenBenchmarking.orgInferences Per Second, More Is BetterONNX Runtime 1.14Model: GPT-2 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching50100150200250SE +/- 0.03, N = 3SE +/- 4.25, N = 15SE +/- 9.23, N = 15103.18132.72218.691. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

OpenVINO

MinAvgMaxEPYC 9654 2P59595651EPYC 9554 2P48610665Xeon Max 9480 2P, HBM Caching164545651OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Age Gender Recognition Retail 0013 FP16-INT8 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching0.3150.630.9451.261.575SE +/- 0.00, N = 3SE +/- 0.00, N = 3SE +/- 0.04, N = 120.990.881.40MIN: 0.85 / MAX: 37.79MIN: 0.83 / MAX: 24.85MIN: 0.7 / MAX: 123.261. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Age Gender Recognition Retail 0013 FP16-INT8 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching30K60K90K120K150KSE +/- 611.95, N = 3SE +/- 574.97, N = 3SE +/- 1694.26, N = 12124618.32128593.2459457.141. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

TensorFlow

MinAvgMaxEPYC 9654 2P57.3328.9406.8EPYC 9554 2P47.7281.8351.3Xeon Max 9480 2P, HBM Caching265.6510.5597.2OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12CPU Power Consumption Monitor160320480640800

OpenBenchmarking.orgimages/sec Per Watt, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 16 - Model: GoogLeNetEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching0.07540.15080.22620.30160.3770.1960.3350.193

OpenBenchmarking.orgimages/sec, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 16 - Model: GoogLeNetEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching20406080100SE +/- 1.21, N = 15SE +/- 0.91, N = 15SE +/- 0.98, N = 364.4994.4198.43

MinAvgMaxEPYC 9654 2P91.1320.9424.6EPYC 9554 2P47.1239.6360.5Xeon Max 9480 2P, HBM Caching260.2466.3587.5OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12CPU Power Consumption Monitor160320480640800

OpenBenchmarking.orgimages/sec Per Watt, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 16 - Model: AlexNetEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching0.24210.48420.72630.96841.21050.5181.0760.493

OpenBenchmarking.orgimages/sec, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 16 - Model: AlexNetEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching60120180240300SE +/- 2.70, N = 15SE +/- 3.19, N = 15SE +/- 1.10, N = 5166.20257.94230.11

MinAvgMaxEPYC 9654 2P104427498EPYC 9554 2P47426517Xeon Max 9480 2P, HBM Caching265554662OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12CPU Power Consumption Monitor2004006008001000

OpenBenchmarking.orgimages/sec Per Watt, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 16 - Model: VGG-16EPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching0.03170.06340.09510.12680.15850.1110.1410.038

OpenBenchmarking.orgimages/sec, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 16 - Model: VGG-16EPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching1326395265SE +/- 0.39, N = 3SE +/- 0.59, N = 3SE +/- 0.61, N = 1547.2760.3121.09

oneDNN

MinAvgMaxEPYC 9654 2P104414522EPYC 9554 2P47365577Xeon Max 9480 2P, HBM Caching266573647OpenBenchmarking.orgWatts, Fewer Is BetteroneDNN 3.1CPU Power Consumption Monitor2004006008001000

OpenBenchmarking.orgms, Fewer Is BetteroneDNN 3.1Harness: Recurrent Neural Network Training - Data Type: bf16bf16bf16 - Engine: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching3K6K9K12K15KSE +/- 18.00, N = 3SE +/- 13.50, N = 5SE +/- 507.18, N = 102492.691305.6615294.60MIN: 7750.751. (CXX) g++ options: -O3 -march=native -fopenmp -msse4.1 -fPIC -pie -ldl -lpthread

MinAvgMaxEPYC 9654 2P106424558EPYC 9554 2P47388564Xeon Max 9480 2P, HBM Caching138534633OpenBenchmarking.orgWatts, Fewer Is BetteroneDNN 3.1CPU Power Consumption Monitor2004006008001000

OpenBenchmarking.orgms, Fewer Is BetteroneDNN 3.1Harness: Deconvolution Batch shapes_1d - Data Type: bf16bf16bf16 - Engine: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching0.571.141.712.282.85SE +/- 0.004515, N = 3SE +/- 0.003878, N = 3SE +/- 0.009112, N = 152.5332601.7616200.483075MIN: 1.7MIN: 1.591. (CXX) g++ options: -O3 -march=native -fopenmp -msse4.1 -fPIC -pie -ldl -lpthread

MinAvgMaxEPYC 9654 2P90348491EPYC 9554 2P47267375Xeon Max 9480 2P, HBM Caching260530626OpenBenchmarking.orgWatts, Fewer Is BetteroneDNN 3.1CPU Power Consumption Monitor2004006008001000

OpenBenchmarking.orgms, Fewer Is BetteroneDNN 3.1Harness: IP Shapes 1D - Data Type: bf16bf16bf16 - Engine: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching48121620SE +/- 0.81790, N = 15SE +/- 0.95058, N = 12SE +/- 0.22497, N = 1516.9823011.196697.88016MIN: 8.09MIN: 5.64MIN: 4.451. (CXX) g++ options: -O3 -march=native -fopenmp -msse4.1 -fPIC -pie -ldl -lpthread

101 Results Shown

oneDNN
OpenVINO
oneDNN
OpenVINO
TensorFlow
OpenVINO
oneDNN
OpenVINO
TensorFlow:
  CPU - 512 - AlexNet
  CPU - 512 - GoogLeNet
  CPU - 512 - ResNet-50
libxsmm
OpenVINO:
  Vehicle Detection FP16 - CPU
  Face Detection FP16-INT8 - CPU
  Weld Porosity Detection FP16 - CPU
  Face Detection FP16 - CPU
oneDNN
OpenVINO:
  Vehicle Detection FP16-INT8 - CPU
  Weld Porosity Detection FP16-INT8 - CPU
  Weld Porosity Detection FP16 - CPU
  Face Detection FP16-INT8 - CPU
  Machine Translation EN To DE FP16 - CPU
TensorFlow
OpenVINO:
  Age Gender Recognition Retail 0013 FP16 - CPU
  Face Detection FP16 - CPU
  Person Vehicle Bike Detection FP16 - CPU
  Person Detection FP32 - CPU
  Person Detection FP16 - CPU
  Person Detection FP16 - CPU
  Person Detection FP32 - CPU
  Age Gender Recognition Retail 0013 FP16 - CPU
ONNX Runtime
OpenVINO
libxsmm:
  64
  32
  256
System Power Consumption Monitor:
  Phoronix Test Suite System Monitoring:
    Watts
    Megahertz
  System Power Consumption Monitor:
    Watts
  CPU Peak Freq (Highest CPU Core Frequency) Monitor:
    Megahertz
  System Power Consumption Monitor:
    Watts
  CPU Peak Freq (Highest CPU Core Frequency) Monitor:
    Megahertz
  System Power Consumption Monitor:
    Watts
  CPU Peak Freq (Highest CPU Core Frequency) Monitor:
    Megahertz
  System Power Consumption Monitor:
    Watts
  CPU Peak Freq (Highest CPU Core Frequency) Monitor:
    Megahertz
  System Power Consumption Monitor:
    Watts
  CPU Peak Freq (Highest CPU Core Frequency) Monitor:
    Megahertz
  System Power Consumption Monitor:
    Watts
  CPU Peak Freq (Highest CPU Core Frequency) Monitor:
    Megahertz
  System Power Consumption Monitor:
    Watts
  CPU Peak Freq (Highest CPU Core Frequency) Monitor:
    Megahertz
  System Power Consumption Monitor:
    Watts
  CPU Peak Freq (Highest CPU Core Frequency) Monitor:
    Megahertz
  System Power Consumption Monitor:
    Watts
  CPU Peak Freq (Highest CPU Core Frequency) Monitor:
    Megahertz
  System Power Consumption Monitor:
    Watts
  CPU Peak Freq (Highest CPU Core Frequency) Monitor:
    Megahertz
  System Power Consumption Monitor:
    Watts
  CPU Peak Freq (Highest CPU Core Frequency) Monitor:
    Megahertz
  System Power Consumption Monitor:
    Watts
  CPU Peak Freq (Highest CPU Core Frequency) Monitor:
    Megahertz
  System Power Consumption Monitor:
    Watts
  CPU Peak Freq (Highest CPU Core Frequency) Monitor:
    Megahertz
  Phoronix Test Suite System Monitoring:
    Watts
  CPU Power Consumption Monitor:
    Watts
ONNX Runtime:
  ResNet50 v1-12-int8 - CPU - Standard:
    Inference Time Cost (ms)
    Inferences Per Second
ONNX Runtime
ONNX Runtime:
  fcn-resnet101-11 - CPU - Standard:
    Inference Time Cost (ms)
    Inferences Per Second
ONNX Runtime
ONNX Runtime:
  CaffeNet 12-int8 - CPU - Standard:
    Inference Time Cost (ms)
    Inferences Per Second
ONNX Runtime
ONNX Runtime:
  bertsquad-12 - CPU - Standard:
    Inference Time Cost (ms)
    Inferences Per Second
ONNX Runtime
ONNX Runtime:
  yolov4 - CPU - Standard:
    Inference Time Cost (ms)
    Inferences Per Second
ONNX Runtime
ONNX Runtime:
  GPT-2 - CPU - Standard:
    Inference Time Cost (ms)
    Inferences Per Second
OpenVINO
OpenVINO:
  Age Gender Recognition Retail 0013 FP16-INT8 - CPU:
    ms
    FPS
TensorFlow:
  CPU Power Consumption Monitor
  CPU - 16 - GoogLeNet
TensorFlow
TensorFlow:
  CPU Power Consumption Monitor
  CPU - 16 - AlexNet
TensorFlow
TensorFlow:
  CPU Power Consumption Monitor
  CPU - 16 - VGG-16
TensorFlow
oneDNN
oneDNN
oneDNN
oneDNN
oneDNN
oneDNN