Xeon Max AMX HBM2e

Benchmarks for a future article on Phoronix.

Compare your own system(s) to this result file with the Phoronix Test Suite by running the command: phoronix-test-suite benchmark 2306302-NE-XEONMAXAM96
Jump To Table - Results

View

Do Not Show Noisy Results
Do Not Show Results With Incomplete Data
Do Not Show Results With Little Change/Spread
List Notable Results
Show Result Confidence Charts

Limit displaying results to tests within:

CPU Massive 2 Tests
Creator Workloads 2 Tests
HPC - High Performance Computing 4 Tests
Machine Learning 4 Tests
Multi-Core 2 Tests
Intel oneAPI 2 Tests
Python Tests 3 Tests

Statistics

Show Overall Harmonic Mean(s)
Show Overall Geometric Mean
Show Geometric Means Per-Suite/Category
Show Wins / Losses Counts (Pie Chart)
Normalize Results
Remove Outliers Before Calculating Averages

Graph Settings

Force Line Graphs Where Applicable
Convert To Scalar Where Applicable
Disable Color Branding
Prefer Vertical Bar Graphs
No Box Plots
On Line Graphs With Missing Data, Connect The Line Gaps

Additional Graphs

Show Perf Per Core/Thread Calculation Graphs Where Applicable
Show Perf Per Clock Calculation Graphs Where Applicable

Multi-Way Comparison

Condense Multi-Option Tests Into Single Result Graphs
Condense Test Profiles With Multiple Version Results Into Single Result Graphs

Table

Show Detailed System Result Table

Run Management

Highlight
Result
Hide
Result
Result
Identifier
Performance Per
Dollar
Date
Run
  Test
  Duration
EPYC 9654 2P
June 28 2023
  6 Hours, 59 Minutes
EPYC 9554 2P
June 28 2023
  5 Hours, 42 Minutes
Xeon Max 9480 2P, HBM Caching
June 29 2023
  13 Hours, 47 Minutes
Invert Hiding All Results Option
  8 Hours, 49 Minutes

Only show results where is faster than
Only show results matching title/arguments (delimit multiple options with a comma):
Do not show results matching title/arguments (delimit multiple options with a comma):

EPYC 9654 2P

Processor: 2 x AMD EPYC 9654 96-Core @ 2.40GHz (192 Cores / 384 Threads), Motherboard: AMD Titanite_4G (RTI1007B BIOS), Chipset: AMD Device 14a4, Memory: 1520GB, Disk: 7682GB INTEL SSDPF2KX076TZ, Graphics: ASPEED, Monitor: VGA HDMI, Network: Broadcom NetXtreme BCM5720 PCIe

OS: Ubuntu 23.04, Kernel: 6.2.0-20-generic (x86_64), Desktop: GNOME Shell 44.0, Display Server: X Server 1.21.1.7, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1080

Kernel Notes: Transparent Huge Pages: madvise
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa101121
Python Notes: Python 3.11.2
Security Notes: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected

EPYC 9554 2P

Changed Processor to 2 x AMD EPYC 9554 64-Core @ 3.10GHz (128 Cores / 256 Threads).

Xeon Max 9480 2P, HBM Caching

Processor: 2 x Intel Xeon Max 9480 @ 3.50GHz (112 Cores / 224 Threads), Motherboard: Supermicro X13DEM v1.10 (1.3 BIOS), Chipset: Intel Device 1bce, Memory: 512GB, Disk: 7682GB INTEL SSDPF2KX076TZ, Graphics: ASPEED, Monitor: VE228, Network: 2 x Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb

Kernel Notes: Transparent Huge Pages: madvise
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c0001d1
Python Notes: Python 3.11.2
Security Notes: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

Xeon Max AMX HBM2eProcessorMotherboardChipsetMemoryDiskGraphicsMonitorNetworkOSKernelDesktopDisplay ServerCompilerFile-SystemScreen ResolutionEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching2 x AMD EPYC 9654 96-Core @ 2.40GHz (192 Cores / 384 Threads)AMD Titanite_4G (RTI1007B BIOS)AMD Device 14a41520GB7682GB INTEL SSDPF2KX076TZASPEEDVGA HDMIBroadcom NetXtreme BCM5720 PCIeUbuntu 23.046.2.0-20-generic (x86_64)GNOME Shell 44.0X Server 1.21.1.7GCC 12.2.0ext41920x10802 x AMD EPYC 9554 64-Core @ 3.10GHz (128 Cores / 256 Threads)2 x Intel Xeon Max 9480 @ 3.50GHz (112 Cores / 224 Threads)Supermicro X13DEM v1.10 (1.3 BIOS)Intel Device 1bce512GBVE2282 x Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200GbOpenBenchmarking.orgKernel Details- Transparent Huge Pages: madviseCompiler Details- --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details- EPYC 9654 2P: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa101121- EPYC 9554 2P: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa101121- Xeon Max 9480 2P, HBM Caching: Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c0001d1Python Details- Python 3.11.2Security Details- EPYC 9654 2P: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected - EPYC 9554 2P: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected - Xeon Max 9480 2P, HBM Caching: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

EPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM CachingResult OverviewPhoronix Test Suite100%171%242%313%384%oneDNNlibxsmmTensorFlowONNX RuntimeOpenVINO

EPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM CachingPer Watt Result OverviewPhoronix Test Suite100%168%236%303%TensorFlowTensorFlowTensorFlowTensorFlowTensorFlowTensorFlowTensorFlowTensorFlowlibxsmmCPU - 512 - ResNet-50CPU - 512 - GoogLeNetCPU - 16 - ResNet-50CPU - 16 - GoogLeNetCPU - 512 - AlexNetCPU - 512 - VGG-16CPU - 16 - AlexNetCPU - 16 - VGG-16128

Xeon Max AMX HBM2etensorflow: CPU - 512 - VGG-16libxsmm: 128tensorflow: CPU - 512 - ResNet-50onnx: bertsquad-12 - CPU - Standardonnx: bertsquad-12 - CPU - Standardlibxsmm: 256onnx: CaffeNet 12-int8 - CPU - Standardonnx: CaffeNet 12-int8 - CPU - Standardonnx: GPT-2 - CPU - Standardonnx: GPT-2 - CPU - Standardonnx: ResNet50 v1-12-int8 - CPU - Standardonnx: ResNet50 v1-12-int8 - CPU - Standardonednn: Recurrent Neural Network Inference - bf16bf16bf16 - CPUonednn: Recurrent Neural Network Training - bf16bf16bf16 - CPUonnx: fcn-resnet101-11 - CPU - Standardonnx: fcn-resnet101-11 - CPU - Standardonnx: ArcFace ResNet-100 - CPU - Standardonnx: ArcFace ResNet-100 - CPU - Standardonnx: yolov4 - CPU - Standardonnx: yolov4 - CPU - Standardtensorflow: CPU - 16 - VGG-16tensorflow: CPU - 512 - GoogLeNetopenvino: Vehicle Detection FP16 - CPUopenvino: Vehicle Detection FP16 - CPUopenvino: Age Gender Recognition Retail 0013 FP16-INT8 - CPUopenvino: Age Gender Recognition Retail 0013 FP16-INT8 - CPUtensorflow: CPU - 16 - GoogLeNetopenvino: Face Detection FP16-INT8 - CPUopenvino: Face Detection FP16-INT8 - CPUonednn: IP Shapes 1D - bf16bf16bf16 - CPUopenvino: Face Detection FP16 - CPUopenvino: Face Detection FP16 - CPUopenvino: Person Detection FP16 - CPUopenvino: Person Detection FP16 - CPUopenvino: Person Vehicle Bike Detection FP16 - CPUopenvino: Person Vehicle Bike Detection FP16 - CPUopenvino: Person Detection FP32 - CPUopenvino: Person Detection FP32 - CPUtensorflow: CPU - 512 - AlexNetopenvino: Machine Translation EN To DE FP16 - CPUopenvino: Machine Translation EN To DE FP16 - CPUopenvino: Vehicle Detection FP16-INT8 - CPUopenvino: Vehicle Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUopenvino: Age Gender Recognition Retail 0013 FP16 - CPUopenvino: Age Gender Recognition Retail 0013 FP16 - CPUopenvino: Weld Porosity Detection FP16 - CPUopenvino: Weld Porosity Detection FP16 - CPUtensorflow: CPU - 16 - ResNet-50onednn: Deconvolution Batch shapes_1d - bf16bf16bf16 - CPUtensorflow: CPU - 16 - AlexNetonednn: IP Shapes 3D - bf16bf16bf16 - CPUonednn: Convolution Batch Shapes Auto - bf16bf16bf16 - CPUlibxsmm: 64libxsmm: 32onednn: Deconvolution Batch shapes_3d - bf16bf16bf16 - CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching120.623426.1171.5698.995710.179444379.62.30611435.1949.68904103.1846.87053146.5552790.902492.69212.8634.6979751.789819.3084204.0914.9003347.27535.526.497390.970.99124618.3264.49250.39191.4116.9823471.82101.561092.9743.645.259133.221098.9643.401757.1650.42950.944.3610985.179.9519170.610.55146867.484.869843.0325.252.53326166.203.802090.4610532270.21298.00.732100109.894112.9188.1785.900911.68714603.32.01949495.2407.64735132.7216.55989152.439925.7671305.66214.6064.6598241.644524.0122195.8995.1050560.31593.205.345981.570.88128593.2494.41205.71155.2811.19669389.5382.04832.0238.244.297447.60825.5238.531813.6840.67786.013.549010.458.0915659.320.52162207.973.978039.5733.401.76162257.942.205960.3113782692.71402.20.45727230.191814.376.9387.763811.45761.78296562.2184.70849218.6928.02166125.2841325.4515294.6191.6235.3351442.180123.7729130.3707.7004621.09236.4412.072315.121.4059457.1498.43333.51334.897.88016232.01120.40834.9733.4017.836265.77819.7633.97718.5447.29590.9220.235528.033.4730938.500.44108515.126.4416941.9940.150.483075230.1191.98143.673563.53525OpenBenchmarking.org

TensorFlow

This is a benchmark of the TensorFlow deep learning framework using the TensorFlow reference benchmarks (tensorflow/benchmarks with tf_cnn_benchmarks.py). Note with the Phoronix Test Suite there is also pts/tensorflow-lite for benchmarking the TensorFlow Lite binaries if desired for complementary metrics. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgimages/sec, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 512 - Model: VGG-16Xeon Max 9480 2P, HBM CachingEPYC 9554 2PEPYC 9654 2P306090120150SE +/- 0.33, N = 5SE +/- 1.46, N = 3SE +/- 0.08, N = 330.19109.89120.62
OpenBenchmarking.orgimages/sec Per Watt, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 512 - Model: VGG-16Xeon Max 9480 2P, HBM CachingEPYC 9554 2PEPYC 9654 2P0.04410.08820.13230.17640.22050.0560.1840.196
OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12CPU Power Consumption MonitorEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching160320480640800Min: 95.79 / Avg: 616.66 / Max: 655.38Min: 47.29 / Avg: 597.07 / Max: 647.99Min: 163.29 / Avg: 539.75 / Max: 899.871. EPYC 9654 2P: Approximate power consumption of 291885 Joules per run.2. EPYC 9554 2P: Approximate power consumption of 309285 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 1002533 Joules per run.
OpenBenchmarking.orgMegahertz, More Is BetterTensorFlow 2.12CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, HBM Caching6001200180024003000Min: 1671 / Avg: 3334.65 / Max: 3571
OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12System Power Consumption MonitorXeon Max 9480 2P, HBM Caching160320480640800Min: 275 / Avg: 740.43 / Max: 9071. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 1378229 Joules per run.

libxsmm

Libxsmm is an open-source library for specialized dense and sparse matrix operations and deep learning primitives. Libxsmm supports making use of Intel AMX, AVX-512, and other modern CPU instruction set capabilities. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgGFLOPS/s, More Is Betterlibxsmm 2-1.17-3645M N K: 128Xeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P9001800270036004500SE +/- 22.78, N = 9SE +/- 29.11, N = 3SE +/- 21.03, N = 31814.33426.14112.91. (CXX) g++ options: -dynamic -Bstatic -static-libgcc -lgomp -lm -lrt -ldl -lquadmath -lstdc++ -pthread -fPIC -std=c++14 -O2 -fopenmp-simd -funroll-loops -ftree-vectorize -fdata-sections -ffunction-sections -fvisibility=hidden -msse4.2
OpenBenchmarking.orgGFLOPS/s Per Watt, More Is Betterlibxsmm 2-1.17-3645M N K: 128Xeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P36912153.2097.76610.285
OpenBenchmarking.orgWatts, Fewer Is Betterlibxsmm 2-1.17-3645CPU Power Consumption MonitorXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P120240360480600Min: 256.12 / Avg: 565.34 / Max: 625.18Min: 99.14 / Avg: 441.18 / Max: 632.27Min: 47.03 / Avg: 399.88 / Max: 668.241. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 387511 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 365441 Joules per run.3. EPYC 9554 2P: Approximate power consumption of 286182 Joules per run.
OpenBenchmarking.orgMegahertz, More Is Betterlibxsmm 2-1.17-3645CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, HBM Caching6001200180024003000Min: 1122 / Avg: 3487.45 / Max: 3507
OpenBenchmarking.orgWatts, Fewer Is Betterlibxsmm 2-1.17-3645System Power Consumption MonitorXeon Max 9480 2P, HBM Caching150300450600750Min: 395 / Avg: 776.46 / Max: 8321. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 533258 Joules per run.

TensorFlow

This is a benchmark of the TensorFlow deep learning framework using the TensorFlow reference benchmarks (tensorflow/benchmarks with tf_cnn_benchmarks.py). Note with the Phoronix Test Suite there is also pts/tensorflow-lite for benchmarking the TensorFlow Lite binaries if desired for complementary metrics. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgimages/sec, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 512 - Model: ResNet-50Xeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P4080120160200SE +/- 0.66, N = 3SE +/- 0.26, N = 3SE +/- 0.70, N = 376.93171.56188.17
OpenBenchmarking.orgimages/sec Per Watt, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 512 - Model: ResNet-50Xeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P0.08710.17420.26130.34840.43550.1360.3410.387
OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12CPU Power Consumption MonitorXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P130260390520650Min: 230.66 / Avg: 564.58 / Max: 721.21Min: 58.1 / Avg: 502.68 / Max: 561.9Min: 48.93 / Avg: 486.26 / Max: 556.881. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 428896 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 171080 Joules per run.3. EPYC 9554 2P: Approximate power consumption of 151225 Joules per run.
OpenBenchmarking.orgMegahertz, More Is BetterTensorFlow 2.12CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, HBM Caching6001200180024003000Min: 1900 / Avg: 3481.66 / Max: 3577
OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12System Power Consumption MonitorXeon Max 9480 2P, HBM Caching150300450600750Min: 401 / Avg: 775.8 / Max: 8501. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 590643 Joules per run.

ONNX Runtime

ONNX Runtime is developed by Microsoft and partners as a open-source, cross-platform, high performance machine learning inferencing and training accelerator. This test profile runs the ONNX Runtime with various models available from the ONNX Model Zoo. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: bertsquad-12 - Device: CPU - Executor: StandardEPYC 9654 2PXeon Max 9480 2P, HBM CachingEPYC 9554 2P20406080100SE +/- 2.35, N = 15SE +/- 1.73, N = 15SE +/- 1.46, N = 1599.0087.7685.901. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt
OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14CPU Power Consumption MonitorXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P110220330440550Min: 264.43 / Avg: 572.92 / Max: 616.32Min: 57.25 / Avg: 394.67 / Max: 418.06Min: 47.62 / Avg: 368.65 / Max: 387.581. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 54160 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 37152 Joules per run.3. EPYC 9554 2P: Approximate power consumption of 34727 Joules per run.
OpenBenchmarking.orgMegahertz, More Is BetterONNX Runtime 1.14CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, HBM Caching6001200180024003000Min: 3111 / Avg: 3499.75 / Max: 3508
OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14System Power Consumption MonitorXeon Max 9480 2P, HBM Caching150300450600750Min: 399 / Avg: 783.38 / Max: 8281. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 74264 Joules per run.

OpenBenchmarking.orgInferences Per Second, More Is BetterONNX Runtime 1.14Model: bertsquad-12 - Device: CPU - Executor: StandardEPYC 9654 2PXeon Max 9480 2P, HBM CachingEPYC 9554 2P3691215SE +/- 0.24, N = 15SE +/- 0.23, N = 15SE +/- 0.19, N = 1510.1811.4611.691. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt
OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: bertsquad-12 - Device: CPU - Executor: StandardEPYC 9654 2PXeon Max 9480 2P, HBM CachingEPYC 9554 2P20406080100SE +/- 2.35, N = 15SE +/- 1.73, N = 15SE +/- 1.46, N = 1599.0087.7685.901. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

libxsmm

Libxsmm is an open-source library for specialized dense and sparse matrix operations and deep learning primitives. Libxsmm supports making use of Intel AMX, AVX-512, and other modern CPU instruction set capabilities. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgGFLOPS/s, More Is Betterlibxsmm 2-1.17-3645M N K: 256EPYC 9654 2PEPYC 9554 2P10002000300040005000SE +/- 47.44, N = 5SE +/- 34.70, N = 34379.64603.31. (CXX) g++ options: -dynamic -Bstatic -static-libgcc -lgomp -lm -lrt -ldl -lquadmath -lstdc++ -pthread -fPIC -std=c++14 -O2 -fopenmp-simd -funroll-loops -ftree-vectorize -fdata-sections -ffunction-sections -fvisibility=hidden -msse4.2
OpenBenchmarking.orgGFLOPS/s Per Watt, More Is Betterlibxsmm 2-1.17-3645M N K: 256EPYC 9654 2PEPYC 9554 2P369121510.0711.67
OpenBenchmarking.orgWatts, Fewer Is Betterlibxsmm 2-1.17-3645CPU Power Consumption MonitorEPYC 9654 2PEPYC 9554 2P120240360480600Min: 89.89 / Avg: 434.8 / Max: 664.56Min: 47.99 / Avg: 394.51 / Max: 694.951. EPYC 9654 2P: Approximate power consumption of 128179 Joules per run.2. EPYC 9554 2P: Approximate power consumption of 97312 Joules per run.

ONNX Runtime

ONNX Runtime is developed by Microsoft and partners as a open-source, cross-platform, high performance machine learning inferencing and training accelerator. This test profile runs the ONNX Runtime with various models available from the ONNX Model Zoo. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: CaffeNet 12-int8 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching0.51891.03781.55672.07562.5945SE +/- 0.03734, N = 15SE +/- 0.02472, N = 4SE +/- 0.02850, N = 152.306112.019491.782961. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt
OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14CPU Power Consumption MonitorXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P110220330440550Min: 261.61 / Avg: 573.75 / Max: 607.12Min: 56.74 / Avg: 454.2 / Max: 477.67Min: 47.45 / Avg: 357.73 / Max: 374.811. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 53817 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 42513 Joules per run.3. EPYC 9554 2P: Approximate power consumption of 33537 Joules per run.
OpenBenchmarking.orgMegahertz, More Is BetterONNX Runtime 1.14CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, HBM Caching6001200180024003000Min: 3500 / Avg: 3500.01 / Max: 3507
OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14System Power Consumption MonitorXeon Max 9480 2P, HBM Caching150300450600750Min: 410 / Avg: 784.54 / Max: 8441. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 73747 Joules per run.

OpenBenchmarking.orgInferences Per Second, More Is BetterONNX Runtime 1.14Model: CaffeNet 12-int8 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching120240360480600SE +/- 7.41, N = 15SE +/- 5.94, N = 4SE +/- 8.12, N = 15435.19495.24562.221. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt
OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: CaffeNet 12-int8 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching0.51891.03781.55672.07562.5945SE +/- 0.03734, N = 15SE +/- 0.02472, N = 4SE +/- 0.02850, N = 152.306112.019491.782961. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: GPT-2 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching3691215SE +/- 0.00311, N = 3SE +/- 0.25707, N = 15SE +/- 0.23765, N = 159.689047.647354.708491. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt
OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14CPU Power Consumption MonitorXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P100200300400500Min: 169.41 / Avg: 521.38 / Max: 555.81Min: 58.72 / Avg: 257.53 / Max: 270.63Min: 46.8 / Avg: 221.99 / Max: 233.391. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 49462 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 24379 Joules per run.3. EPYC 9554 2P: Approximate power consumption of 21000 Joules per run.
OpenBenchmarking.orgMegahertz, More Is BetterONNX Runtime 1.14CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, HBM Caching6001200180024003000Min: 3500 / Avg: 3500.97 / Max: 3517
OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14System Power Consumption MonitorXeon Max 9480 2P, HBM Caching140280420560700Min: 281 / Avg: 720.02 / Max: 7701. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 68450 Joules per run.

OpenBenchmarking.orgInferences Per Second, More Is BetterONNX Runtime 1.14Model: GPT-2 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching50100150200250SE +/- 0.03, N = 3SE +/- 4.25, N = 15SE +/- 9.23, N = 15103.18132.72218.691. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt
OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: GPT-2 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching3691215SE +/- 0.00311, N = 3SE +/- 0.25707, N = 15SE +/- 0.23765, N = 159.689047.647354.708491. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: ResNet50 v1-12-int8 - Device: CPU - Executor: StandardXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P246810SE +/- 0.15815, N = 15SE +/- 0.15229, N = 15SE +/- 0.04787, N = 38.021666.870536.559891. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt
OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14CPU Power Consumption MonitorXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P110220330440550Min: 270.52 / Avg: 569.27 / Max: 613.29Min: 31.88 / Avg: 465.24 / Max: 489.61Min: 47.64 / Avg: 368.58 / Max: 386.51. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 53284 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 43547 Joules per run.3. EPYC 9554 2P: Approximate power consumption of 34769 Joules per run.
OpenBenchmarking.orgMegahertz, More Is BetterONNX Runtime 1.14CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, HBM Caching6001200180024003000Min: 2006 / Avg: 3497.11 / Max: 3509
OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14System Power Consumption MonitorXeon Max 9480 2P, HBM Caching150300450600750Min: 410 / Avg: 784.18 / Max: 8301. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 73608 Joules per run.

OpenBenchmarking.orgInferences Per Second, More Is BetterONNX Runtime 1.14Model: ResNet50 v1-12-int8 - Device: CPU - Executor: StandardXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P306090120150SE +/- 2.29, N = 15SE +/- 3.31, N = 15SE +/- 1.11, N = 3125.28146.56152.441. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt
OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: ResNet50 v1-12-int8 - Device: CPU - Executor: StandardXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P246810SE +/- 0.15815, N = 15SE +/- 0.15229, N = 15SE +/- 0.04787, N = 38.021666.870536.559891. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

oneDNN

This is a test of the Intel oneDNN as an Intel-optimized library for Deep Neural Networks and making use of its built-in benchdnn functionality. The result is the total perf time reported. Intel oneDNN was formerly known as DNNL (Deep Neural Network Library) and MKL-DNN before being rebranded as part of the Intel oneAPI toolkit. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgms, Fewer Is BetteroneDNN 3.1Harness: Recurrent Neural Network Inference - Data Type: bf16bf16bf16 - Engine: CPUEPYC 9654 2PXeon Max 9480 2P, HBM CachingEPYC 9554 2P6001200180024003000SE +/- 29.80, N = 15SE +/- 20.64, N = 12SE +/- 2.71, N = 32790.901325.45925.77MIN: 2391.98MIN: 1025.18MIN: 909.671. (CXX) g++ options: -O3 -march=native -fopenmp -msse4.1 -fPIC -pie -ldl -lpthread
MinAvgMaxXeon Max 9480 2P, HBM Caching259561644EPYC 9654 2P96412555EPYC 9554 2P47356602OpenBenchmarking.orgWatts, Fewer Is BetteroneDNN 3.1CPU Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 47136 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 37006 Joules per run.3. EPYC 9554 2P: Approximate power consumption of 29795 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching230027023507OpenBenchmarking.orgMegahertz, More Is BetteroneDNN 3.1CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching402767901OpenBenchmarking.orgWatts, Fewer Is BetteroneDNN 3.1System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 64567 Joules per run.

OpenBenchmarking.orgms, Fewer Is BetteroneDNN 3.1Harness: Recurrent Neural Network Training - Data Type: bf16bf16bf16 - Engine: CPUXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P3K6K9K12K15KSE +/- 507.18, N = 10SE +/- 18.00, N = 3SE +/- 13.50, N = 515294.602492.691305.66MIN: 7750.751. (CXX) g++ options: -O3 -march=native -fopenmp -msse4.1 -fPIC -pie -ldl -lpthread
OpenBenchmarking.orgWatts, Fewer Is BetteroneDNN 3.1CPU Power Consumption MonitorXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P110220330440550Min: 265.69 / Avg: 572.8 / Max: 646.78Min: 103.84 / Avg: 413.83 / Max: 522.24Min: 47.23 / Avg: 365.29 / Max: 577.491. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 91132 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 37245 Joules per run.3. EPYC 9554 2P: Approximate power consumption of 31196 Joules per run.
OpenBenchmarking.orgMegahertz, More Is BetteroneDNN 3.1CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, HBM Caching6001200180024003000Min: 1951 / Avg: 2982.06 / Max: 3514
OpenBenchmarking.orgWatts, Fewer Is BetteroneDNN 3.1System Power Consumption MonitorXeon Max 9480 2P, HBM Caching160320480640800Min: 410 / Avg: 782.09 / Max: 8891. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 124744 Joules per run.

ONNX Runtime

ONNX Runtime is developed by Microsoft and partners as a open-source, cross-platform, high performance machine learning inferencing and training accelerator. This test profile runs the ONNX Runtime with various models available from the ONNX Model Zoo. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: fcn-resnet101-11 - Device: CPU - Executor: StandardEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM Caching50100150200250SE +/- 1.01, N = 3SE +/- 1.01, N = 3SE +/- 8.54, N = 15214.61212.86191.621. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt
OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14CPU Power Consumption MonitorXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P120240360480600Min: 244.09 / Avg: 568.72 / Max: 664.63Min: 57.11 / Avg: 514.32 / Max: 545.61Min: 46.77 / Avg: 420.17 / Max: 447.271. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 53650 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 48346 Joules per run.3. EPYC 9554 2P: Approximate power consumption of 39636 Joules per run.
OpenBenchmarking.orgMegahertz, More Is BetterONNX Runtime 1.14CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, HBM Caching6001200180024003000Min: 1900 / Avg: 3482.84 / Max: 3511
OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14System Power Consumption MonitorXeon Max 9480 2P, HBM Caching150300450600750Min: 400 / Avg: 778.48 / Max: 8531. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 73540 Joules per run.

OpenBenchmarking.orgInferences Per Second, More Is BetterONNX Runtime 1.14Model: fcn-resnet101-11 - Device: CPU - Executor: StandardEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM Caching1.20042.40083.60124.80166.002SE +/- 0.02211, N = 3SE +/- 0.02211, N = 3SE +/- 0.18914, N = 154.659824.697975.335141. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt
OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: fcn-resnet101-11 - Device: CPU - Executor: StandardEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM Caching50100150200250SE +/- 1.01, N = 3SE +/- 1.01, N = 3SE +/- 8.54, N = 15214.61212.86191.621. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: ArcFace ResNet-100 - Device: CPU - Executor: StandardEPYC 9654 2PXeon Max 9480 2P, HBM CachingEPYC 9554 2P1224364860SE +/- 0.23, N = 3SE +/- 0.60, N = 15SE +/- 0.16, N = 351.7942.1841.641. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt
OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14CPU Power Consumption MonitorXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P110220330440550Min: 261.02 / Avg: 569.95 / Max: 612.98Min: 58.43 / Avg: 464.94 / Max: 493.74Min: 47.34 / Avg: 373.5 / Max: 394.091. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 53765 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 43705 Joules per run.3. EPYC 9554 2P: Approximate power consumption of 35109 Joules per run.
OpenBenchmarking.orgMegahertz, More Is BetterONNX Runtime 1.14CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, HBM Caching6001200180024003000Min: 3500 / Avg: 3500.03 / Max: 3508
OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14System Power Consumption MonitorXeon Max 9480 2P, HBM Caching140280420560700Min: 405 / Avg: 783.58 / Max: 8211. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 74126 Joules per run.

OpenBenchmarking.orgInferences Per Second, More Is BetterONNX Runtime 1.14Model: ArcFace ResNet-100 - Device: CPU - Executor: StandardEPYC 9654 2PXeon Max 9480 2P, HBM CachingEPYC 9554 2P612182430SE +/- 0.09, N = 3SE +/- 0.34, N = 15SE +/- 0.09, N = 319.3123.7724.011. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt
OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: ArcFace ResNet-100 - Device: CPU - Executor: StandardEPYC 9654 2PXeon Max 9480 2P, HBM CachingEPYC 9554 2P1224364860SE +/- 0.23, N = 3SE +/- 0.60, N = 15SE +/- 0.16, N = 351.7942.1841.641. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: yolov4 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching4080120160200SE +/- 1.68, N = 3SE +/- 1.35, N = 3SE +/- 2.64, N = 12204.09195.90130.371. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt
MinAvgMaxXeon Max 9480 2P, HBM Caching262573626EPYC 9654 2P57456482EPYC 9554 2P47360380OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14CPU Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 53984 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 43132 Joules per run.3. EPYC 9554 2P: Approximate power consumption of 34123 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching350035003507OpenBenchmarking.orgMegahertz, More Is BetterONNX Runtime 1.14CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching390783827OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 73942 Joules per run.

OpenBenchmarking.orgInferences Per Second, More Is BetterONNX Runtime 1.14Model: yolov4 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching246810SE +/- 0.04001, N = 3SE +/- 0.03523, N = 3SE +/- 0.13594, N = 124.900335.105057.700461. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt
OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: yolov4 - Device: CPU - Executor: StandardEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching4080120160200SE +/- 1.68, N = 3SE +/- 1.35, N = 3SE +/- 2.64, N = 12204.09195.90130.371. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

TensorFlow

This is a benchmark of the TensorFlow deep learning framework using the TensorFlow reference benchmarks (tensorflow/benchmarks with tf_cnn_benchmarks.py). Note with the Phoronix Test Suite there is also pts/tensorflow-lite for benchmarking the TensorFlow Lite binaries if desired for complementary metrics. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgimages/sec, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 16 - Model: VGG-16Xeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P1326395265SE +/- 0.61, N = 15SE +/- 0.39, N = 3SE +/- 0.59, N = 321.0947.2760.31
OpenBenchmarking.orgimages/sec Per Watt, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 16 - Model: VGG-16Xeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P0.03170.06340.09510.12680.15850.0380.1110.141
OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12CPU Power Consumption MonitorXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P120240360480600Min: 265.14 / Avg: 554.37 / Max: 661.96Min: 104.14 / Avg: 426.73 / Max: 498.21Min: 47.38 / Avg: 426.37 / Max: 516.761. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 50817 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 19060 Joules per run.3. EPYC 9554 2P: Approximate power consumption of 15349 Joules per run.
OpenBenchmarking.orgMegahertz, More Is BetterTensorFlow 2.12CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, HBM Caching6001200180024003000Min: 2600 / Avg: 3476.25 / Max: 3555
OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12System Power Consumption MonitorXeon Max 9480 2P, HBM Caching140280420560700Min: 410 / Avg: 755.75 / Max: 8221. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 69377 Joules per run.

OpenBenchmarking.orgimages/sec, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 512 - Model: GoogLeNetXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P130260390520650SE +/- 1.79, N = 3SE +/- 0.69, N = 3SE +/- 0.96, N = 3236.44535.52593.20
OpenBenchmarking.orgimages/sec Per Watt, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 512 - Model: GoogLeNetXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P0.27250.5450.81751.091.36250.4211.0201.211
MinAvgMaxXeon Max 9480 2P, HBM Caching275562692EPYC 9654 2P56525568EPYC 9554 2P47490537OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12CPU Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 138872 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 59497 Joules per run.3. EPYC 9554 2P: Approximate power consumption of 50120 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching195833363515OpenBenchmarking.orgMegahertz, More Is BetterTensorFlow 2.12CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching425771850OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 190783 Joules per run.

OpenVINO

This is a test of the Intel OpenVINO, a toolkit around neural networks, using its built-in benchmarking support and analyzing the throughput and latency for various models. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Vehicle Detection FP16 - Device: CPUXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P3691215SE +/- 0.08, N = 13SE +/- 0.00, N = 3SE +/- 0.00, N = 312.076.495.34MIN: 7.38 / MAX: 250.92MIN: 5.03 / MAX: 55.65MIN: 4.89 / MAX: 34.71. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF
MinAvgMaxEPYC 9654 2P58636689EPYC 9554 2P50629675Xeon Max 9480 2P, HBM Caching87548657OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. EPYC 9654 2P: Approximate power consumption of 41344 Joules per run.2. EPYC 9554 2P: Approximate power consumption of 40490 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 36629 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching130017383501OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching272742868OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 49737 Joules per run.

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Vehicle Detection FP16 - Device: CPUXeon Max 9480 2P, HBM CachingEPYC 9554 2PEPYC 9654 2P16003200480064008000SE +/- 17.29, N = 13SE +/- 3.49, N = 3SE +/- 2.12, N = 32315.125981.577390.971. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Age Gender Recognition Retail 0013 FP16-INT8 - Device: CPUXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P0.3150.630.9451.261.575SE +/- 0.04, N = 12SE +/- 0.00, N = 3SE +/- 0.00, N = 31.400.990.88MIN: 0.7 / MAX: 123.26MIN: 0.85 / MAX: 37.79MIN: 0.83 / MAX: 24.851. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF
MinAvgMaxEPYC 9554 2P48610665EPYC 9654 2P59595651Xeon Max 9480 2P, HBM Caching164545651OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. EPYC 9554 2P: Approximate power consumption of 39444 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 38859 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 36125 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching110020993500OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching268742856OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 49351 Joules per run.

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Age Gender Recognition Retail 0013 FP16-INT8 - Device: CPUXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P30K60K90K120K150KSE +/- 1694.26, N = 12SE +/- 611.95, N = 3SE +/- 574.97, N = 359457.14124618.32128593.241. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

TensorFlow

This is a benchmark of the TensorFlow deep learning framework using the TensorFlow reference benchmarks (tensorflow/benchmarks with tf_cnn_benchmarks.py). Note with the Phoronix Test Suite there is also pts/tensorflow-lite for benchmarking the TensorFlow Lite binaries if desired for complementary metrics. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgimages/sec, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 16 - Model: GoogLeNetEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching20406080100SE +/- 1.21, N = 15SE +/- 0.91, N = 15SE +/- 0.98, N = 364.4994.4198.43
OpenBenchmarking.orgimages/sec Per Watt, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 16 - Model: GoogLeNetXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P0.07540.15080.22620.30160.3770.1930.1960.335
MinAvgMaxXeon Max 9480 2P, HBM Caching265.6510.5597.2EPYC 9654 2P57.3328.9406.8EPYC 9554 2P47.7281.8351.3OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12CPU Power Consumption Monitor1603204806408001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 12763 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 11271 Joules per run.3. EPYC 9554 2P: Approximate power consumption of 7213 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching266634893505OpenBenchmarking.orgMegahertz, More Is BetterTensorFlow 2.12CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching405693801OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 17564 Joules per run.

OpenVINO

This is a test of the Intel OpenVINO, a toolkit around neural networks, using its built-in benchmarking support and analyzing the throughput and latency for various models. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P70140210280350SE +/- 0.44, N = 3SE +/- 0.04, N = 3SE +/- 0.22, N = 3333.51250.39205.71MIN: 285.32 / MAX: 516.28MIN: 222.36 / MAX: 303.72MIN: 202.38 / MAX: 252.141. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF
MinAvgMaxEPYC 9654 2P59613687EPYC 9554 2P50606675Xeon Max 9480 2P, HBM Caching168531616OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. EPYC 9654 2P: Approximate power consumption of 42688 Joules per run.2. EPYC 9554 2P: Approximate power consumption of 41215 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 46887 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching170023703505OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching270722818OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 64059 Joules per run.

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM Caching70140210280350SE +/- 0.14, N = 3SE +/- 0.06, N = 3SE +/- 0.45, N = 3155.28191.41334.891. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

oneDNN

This is a test of the Intel oneDNN as an Intel-optimized library for Deep Neural Networks and making use of its built-in benchdnn functionality. The result is the total perf time reported. Intel oneDNN was formerly known as DNNL (Deep Neural Network Library) and MKL-DNN before being rebranded as part of the Intel oneAPI toolkit. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgms, Fewer Is BetteroneDNN 3.1Harness: IP Shapes 1D - Data Type: bf16bf16bf16 - Engine: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching48121620SE +/- 0.81790, N = 15SE +/- 0.95058, N = 12SE +/- 0.22497, N = 1516.9823011.196697.88016MIN: 8.09MIN: 5.64MIN: 4.451. (CXX) g++ options: -O3 -march=native -fopenmp -msse4.1 -fPIC -pie -ldl -lpthread
MinAvgMaxXeon Max 9480 2P, HBM Caching260530626EPYC 9654 2P90348491EPYC 9554 2P47267375OpenBenchmarking.orgWatts, Fewer Is BetteroneDNN 3.1CPU Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 10134 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 6653 Joules per run.3. EPYC 9554 2P: Approximate power consumption of 5123 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching260027973507OpenBenchmarking.orgMegahertz, More Is BetteroneDNN 3.1CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching385721838OpenBenchmarking.orgWatts, Fewer Is BetteroneDNN 3.1System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 13796 Joules per run.

OpenVINO

This is a test of the Intel OpenVINO, a toolkit around neural networks, using its built-in benchmarking support and analyzing the throughput and latency for various models. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Face Detection FP16 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching100200300400500SE +/- 0.93, N = 3SE +/- 0.19, N = 3SE +/- 0.80, N = 3471.82389.53232.01MIN: 431.51 / MAX: 569.3MIN: 380.82 / MAX: 432.57MIN: 138.61 / MAX: 770.451. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF
MinAvgMaxEPYC 9654 2P58646726EPYC 9554 2P49638711Xeon Max 9480 2P, HBM Caching136533628OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. EPYC 9654 2P: Approximate power consumption of 44817 Joules per run.2. EPYC 9554 2P: Approximate power consumption of 43188 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 39944 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching120017403505OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching281724839OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 54275 Joules per run.

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Face Detection FP16 - Device: CPUEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM Caching306090120150SE +/- 0.04, N = 3SE +/- 0.25, N = 3SE +/- 0.41, N = 382.04101.56120.401. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Person Detection FP16 - Device: CPUEPYC 9654 2PXeon Max 9480 2P, HBM CachingEPYC 9554 2P2004006008001000SE +/- 5.11, N = 3SE +/- 6.17, N = 3SE +/- 5.30, N = 31092.97834.97832.02MIN: 825.28 / MAX: 1814.82MIN: 558.93 / MAX: 2258.39MIN: 743.95 / MAX: 1258.171. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF
MinAvgMaxEPYC 9654 2P60608699EPYC 9554 2P49596683Xeon Max 9480 2P, HBM Caching204538618OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. EPYC 9654 2P: Approximate power consumption of 42760 Joules per run.2. EPYC 9554 2P: Approximate power consumption of 41100 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 38946 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching190030233512OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching298736817OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 53264 Joules per run.

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Person Detection FP16 - Device: CPUXeon Max 9480 2P, HBM CachingEPYC 9554 2PEPYC 9654 2P1020304050SE +/- 0.24, N = 3SE +/- 0.23, N = 3SE +/- 0.21, N = 333.4038.2443.641. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Person Vehicle Bike Detection FP16 - Device: CPUXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P48121620SE +/- 0.02, N = 3SE +/- 0.00, N = 3SE +/- 0.01, N = 317.835.254.29MIN: 13.06 / MAX: 147.36MIN: 4.36 / MAX: 23.21MIN: 4.11 / MAX: 21.831. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF
MinAvgMaxEPYC 9654 2P59599664EPYC 9554 2P49586647Xeon Max 9480 2P, HBM Caching176531642OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. EPYC 9654 2P: Approximate power consumption of 39943 Joules per run.2. EPYC 9554 2P: Approximate power consumption of 38464 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 41580 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching160021253502OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching284724867OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 56743 Joules per run.

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Person Vehicle Bike Detection FP16 - Device: CPUXeon Max 9480 2P, HBM CachingEPYC 9554 2PEPYC 9654 2P2K4K6K8K10KSE +/- 8.17, N = 3SE +/- 8.54, N = 3SE +/- 3.38, N = 36265.777447.609133.221. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Person Detection FP32 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching2004006008001000SE +/- 5.33, N = 3SE +/- 1.07, N = 3SE +/- 7.20, N = 31098.96825.52819.76MIN: 825.62 / MAX: 1835.76MIN: 747.05 / MAX: 1238.39MIN: 516.88 / MAX: 2598.391. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF
MinAvgMaxEPYC 9654 2P60606703EPYC 9554 2P49600684Xeon Max 9480 2P, HBM Caching228547624OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. EPYC 9654 2P: Approximate power consumption of 42612 Joules per run.2. EPYC 9554 2P: Approximate power consumption of 41612 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 38131 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching190030743506OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching369748857OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 52353 Joules per run.

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Person Detection FP32 - Device: CPUXeon Max 9480 2P, HBM CachingEPYC 9554 2PEPYC 9654 2P1020304050SE +/- 0.31, N = 3SE +/- 0.04, N = 3SE +/- 0.20, N = 333.9738.5343.401. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

TensorFlow

This is a benchmark of the TensorFlow deep learning framework using the TensorFlow reference benchmarks (tensorflow/benchmarks with tf_cnn_benchmarks.py). Note with the Phoronix Test Suite there is also pts/tensorflow-lite for benchmarking the TensorFlow Lite binaries if desired for complementary metrics. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgimages/sec, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 512 - Model: AlexNetXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P400800120016002000SE +/- 9.04, N = 3SE +/- 16.13, N = 7SE +/- 12.94, N = 3718.541757.161813.68
OpenBenchmarking.orgimages/sec Per Watt, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 512 - Model: AlexNetXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P0.88271.76542.64813.53084.41351.3053.8503.923
MinAvgMaxXeon Max 9480 2P, HBM Caching209.1550.7614.5EPYC 9554 2P48.8462.3567.3EPYC 9654 2P58.3456.4541.6OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12CPU Power Consumption Monitor1603204806408001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 46994 Joules per run.2. EPYC 9554 2P: Approximate power consumption of 17107 Joules per run.3. EPYC 9654 2P: Approximate power consumption of 17409 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching260034743512OpenBenchmarking.orgMegahertz, More Is BetterTensorFlow 2.12CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching307752824OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 64146 Joules per run.

OpenVINO

This is a test of the Intel OpenVINO, a toolkit around neural networks, using its built-in benchmarking support and analyzing the throughput and latency for various models. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Machine Translation EN To DE FP16 - Device: CPUEPYC 9654 2PXeon Max 9480 2P, HBM CachingEPYC 9554 2P1122334455SE +/- 0.04, N = 3SE +/- 0.01, N = 3SE +/- 0.08, N = 350.4247.2940.67MIN: 39.07 / MAX: 280.94MIN: 32.14 / MAX: 577.78MIN: 34.96 / MAX: 124.311. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF
MinAvgMaxEPYC 9654 2P59642706EPYC 9554 2P51622683Xeon Max 9480 2P, HBM Caching165554623OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. EPYC 9654 2P: Approximate power consumption of 42185 Joules per run.2. EPYC 9554 2P: Approximate power consumption of 40668 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 38229 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching170020893501OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching274752852OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 51910 Joules per run.

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Machine Translation EN To DE FP16 - Device: CPUXeon Max 9480 2P, HBM CachingEPYC 9554 2PEPYC 9654 2P2004006008001000SE +/- 0.19, N = 3SE +/- 1.56, N = 3SE +/- 0.77, N = 3590.92786.01950.941. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Vehicle Detection FP16-INT8 - Device: CPUXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P510152025SE +/- 0.09, N = 3SE +/- 0.00, N = 3SE +/- 0.00, N = 320.234.363.54MIN: 12.69 / MAX: 166.76MIN: 3.52 / MAX: 38.49MIN: 3.45 / MAX: 21.71. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF
MinAvgMaxEPYC 9654 2P58600646EPYC 9554 2P49590636Xeon Max 9480 2P, HBM Caching170552614OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. EPYC 9654 2P: Approximate power consumption of 38779 Joules per run.2. EPYC 9554 2P: Approximate power consumption of 37983 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 37922 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching130016853500OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching262755817OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 51849 Joules per run.

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Vehicle Detection FP16-INT8 - Device: CPUXeon Max 9480 2P, HBM CachingEPYC 9554 2PEPYC 9654 2P2K4K6K8K10KSE +/- 25.18, N = 3SE +/- 2.20, N = 3SE +/- 0.53, N = 35528.039010.4510985.171. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching3691215SE +/- 0.00, N = 3SE +/- 0.00, N = 3SE +/- 0.00, N = 39.958.093.47MIN: 8.36 / MAX: 40.39MIN: 7.89 / MAX: 45.39MIN: 2.48 / MAX: 56.571. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF
MinAvgMaxEPYC 9654 2P59632685EPYC 9554 2P49623675Xeon Max 9480 2P, HBM Caching137552614OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. EPYC 9654 2P: Approximate power consumption of 41474 Joules per run.2. EPYC 9554 2P: Approximate power consumption of 40518 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 37176 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching150019023504OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching275752830OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 51165 Joules per run.

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM Caching7K14K21K28K35KSE +/- 3.60, N = 3SE +/- 4.81, N = 3SE +/- 38.32, N = 315659.3219170.6130938.501. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Age Gender Recognition Retail 0013 FP16 - Device: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching0.12380.24760.37140.49520.619SE +/- 0.00, N = 3SE +/- 0.00, N = 3SE +/- 0.00, N = 30.550.520.44MIN: 0.5 / MAX: 45.73MIN: 0.49 / MAX: 50.12MIN: 0.33 / MAX: 42.851. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF
MinAvgMaxEPYC 9654 2P62581643EPYC 9554 2P49580637Xeon Max 9480 2P, HBM Caching265570619OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. EPYC 9654 2P: Approximate power consumption of 38142 Joules per run.2. EPYC 9554 2P: Approximate power consumption of 37918 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 37226 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching190034263501OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching392772837OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 50450 Joules per run.

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Age Gender Recognition Retail 0013 FP16 - Device: CPUXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P30K60K90K120K150KSE +/- 967.90, N = 3SE +/- 1825.86, N = 3SE +/- 685.52, N = 3108515.12146867.48162207.971. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16 - Device: CPUXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P246810SE +/- 0.01, N = 3SE +/- 0.00, N = 3SE +/- 0.00, N = 36.444.863.97MIN: 4.24 / MAX: 49.43MIN: 4.02 / MAX: 36.01MIN: 3.87 / MAX: 17.351. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF
MinAvgMaxEPYC 9654 2P57655705EPYC 9554 2P48648698Xeon Max 9480 2P, HBM Caching163557629OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. EPYC 9654 2P: Approximate power consumption of 42138 Joules per run.2. EPYC 9554 2P: Approximate power consumption of 41467 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 37337 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching170019003500OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching279756840OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 50630 Joules per run.

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16 - Device: CPUEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM Caching4K8K12K16K20KSE +/- 1.22, N = 3SE +/- 2.32, N = 3SE +/- 13.63, N = 38039.579843.0316941.991. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

TensorFlow

This is a benchmark of the TensorFlow deep learning framework using the TensorFlow reference benchmarks (tensorflow/benchmarks with tf_cnn_benchmarks.py). Note with the Phoronix Test Suite there is also pts/tensorflow-lite for benchmarking the TensorFlow Lite binaries if desired for complementary metrics. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgimages/sec, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 16 - Model: ResNet-50EPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching918273645SE +/- 0.31, N = 3SE +/- 0.23, N = 3SE +/- 0.32, N = 325.2533.4040.15
OpenBenchmarking.orgimages/sec Per Watt, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 16 - Model: ResNet-50Xeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P0.02590.05180.07770.10360.12950.0740.0770.115
MinAvgMaxXeon Max 9480 2P, HBM Caching276.7539.1593.1EPYC 9654 2P57.2327.2393.8EPYC 9554 2P46.8291.1331.1OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12CPU Power Consumption Monitor1603204806408001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 28395 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 25519 Joules per run.3. EPYC 9554 2P: Approximate power consumption of 17662 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching350035003510OpenBenchmarking.orgMegahertz, More Is BetterTensorFlow 2.12CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching415733808OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 38625 Joules per run.

oneDNN

This is a test of the Intel oneDNN as an Intel-optimized library for Deep Neural Networks and making use of its built-in benchdnn functionality. The result is the total perf time reported. Intel oneDNN was formerly known as DNNL (Deep Neural Network Library) and MKL-DNN before being rebranded as part of the Intel oneAPI toolkit. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgms, Fewer Is BetteroneDNN 3.1Harness: Deconvolution Batch shapes_1d - Data Type: bf16bf16bf16 - Engine: CPUEPYC 9654 2PEPYC 9554 2PXeon Max 9480 2P, HBM Caching0.571.141.712.282.85SE +/- 0.004515, N = 3SE +/- 0.003878, N = 3SE +/- 0.009112, N = 152.5332601.7616200.483075MIN: 1.7MIN: 1.591. (CXX) g++ options: -O3 -march=native -fopenmp -msse4.1 -fPIC -pie -ldl -lpthread
MinAvgMaxXeon Max 9480 2P, HBM Caching138534633EPYC 9654 2P106424558EPYC 9554 2P47388564OpenBenchmarking.orgWatts, Fewer Is BetteroneDNN 3.1CPU Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 13281 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 10745 Joules per run.3. EPYC 9554 2P: Approximate power consumption of 9960 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching216626513508OpenBenchmarking.orgMegahertz, More Is BetteroneDNN 3.1CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching404727850OpenBenchmarking.orgWatts, Fewer Is BetteroneDNN 3.1System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 18187 Joules per run.

TensorFlow

This is a benchmark of the TensorFlow deep learning framework using the TensorFlow reference benchmarks (tensorflow/benchmarks with tf_cnn_benchmarks.py). Note with the Phoronix Test Suite there is also pts/tensorflow-lite for benchmarking the TensorFlow Lite binaries if desired for complementary metrics. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgimages/sec, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 16 - Model: AlexNetEPYC 9654 2PXeon Max 9480 2P, HBM CachingEPYC 9554 2P60120180240300SE +/- 2.70, N = 15SE +/- 1.10, N = 5SE +/- 3.19, N = 15166.20230.11257.94
OpenBenchmarking.orgimages/sec Per Watt, More Is BetterTensorFlow 2.12Device: CPU - Batch Size: 16 - Model: AlexNetXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P0.24210.48420.72630.96841.21050.4930.5181.076
MinAvgMaxXeon Max 9480 2P, HBM Caching260.2466.3587.5EPYC 9654 2P91.1320.9424.6EPYC 9554 2P47.1239.6360.5OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12CPU Power Consumption Monitor1603204806408001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 6529 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 5264 Joules per run.3. EPYC 9554 2P: Approximate power consumption of 3019 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching260033063511OpenBenchmarking.orgMegahertz, More Is BetterTensorFlow 2.12CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching408636785OpenBenchmarking.orgWatts, Fewer Is BetterTensorFlow 2.12System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 8910 Joules per run.

oneDNN

This is a test of the Intel oneDNN as an Intel-optimized library for Deep Neural Networks and making use of its built-in benchdnn functionality. The result is the total perf time reported. Intel oneDNN was formerly known as DNNL (Deep Neural Network Library) and MKL-DNN before being rebranded as part of the Intel oneAPI toolkit. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgms, Fewer Is BetteroneDNN 3.1Harness: IP Shapes 3D - Data Type: bf16bf16bf16 - Engine: CPUXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P20406080100SE +/- 0.61972, N = 5SE +/- 0.03459, N = 15SE +/- 0.02374, N = 1591.981403.802092.20596MIN: 79.161. (CXX) g++ options: -O3 -march=native -fopenmp -msse4.1 -fPIC -pie -ldl -lpthread
MinAvgMaxXeon Max 9480 2P, HBM Caching260498639EPYC 9654 2P89333497EPYC 9554 2P47256402OpenBenchmarking.orgWatts, Fewer Is BetteroneDNN 3.1CPU Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 6670 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 4392 Joules per run.3. EPYC 9554 2P: Approximate power consumption of 3367 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching260030873508OpenBenchmarking.orgMegahertz, More Is BetteroneDNN 3.1CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching400678830OpenBenchmarking.orgWatts, Fewer Is BetteroneDNN 3.1System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 9085 Joules per run.

OpenBenchmarking.orgms, Fewer Is BetteroneDNN 3.1Harness: Convolution Batch Shapes Auto - Data Type: bf16bf16bf16 - Engine: CPUXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P0.82661.65322.47983.30644.133SE +/- 0.025287, N = 7SE +/- 0.003345, N = 15SE +/- 0.002531, N = 73.6735600.4610530.311378MIN: 3.031. (CXX) g++ options: -O3 -march=native -fopenmp -msse4.1 -fPIC -pie -ldl -lpthread
MinAvgMaxXeon Max 9480 2P, HBM Caching267471667EPYC 9654 2P87337534EPYC 9554 2P47325610OpenBenchmarking.orgWatts, Fewer Is BetteroneDNN 3.1CPU Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 4840 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 3441 Joules per run.3. EPYC 9554 2P: Approximate power consumption of 3295 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching209629513506OpenBenchmarking.orgMegahertz, More Is BetteroneDNN 3.1CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching400642881OpenBenchmarking.orgWatts, Fewer Is BetteroneDNN 3.1System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 6608 Joules per run.

libxsmm

Libxsmm is an open-source library for specialized dense and sparse matrix operations and deep learning primitives. Libxsmm supports making use of Intel AMX, AVX-512, and other modern CPU instruction set capabilities. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgGFLOPS/s, More Is Betterlibxsmm 2-1.17-3645M N K: 64EPYC 9654 2PEPYC 9554 2P6001200180024003000SE +/- 33.30, N = 15SE +/- 5.63, N = 72270.22692.71. (CXX) g++ options: -dynamic -Bstatic -static-libgcc -lgomp -lm -lrt -ldl -lquadmath -lstdc++ -pthread -fPIC -std=c++14 -O2 -fopenmp-simd -funroll-loops -ftree-vectorize -fdata-sections -ffunction-sections -fvisibility=hidden -msse4.2
OpenBenchmarking.orgGFLOPS/s Per Watt, More Is Betterlibxsmm 2-1.17-3645M N K: 64EPYC 9654 2PEPYC 9554 2P2468105.9757.839
MinAvgMaxEPYC 9654 2P98380613EPYC 9554 2P47344635OpenBenchmarking.orgWatts, Fewer Is Betterlibxsmm 2-1.17-3645CPU Power Consumption Monitor20040060080010001. EPYC 9654 2P: Approximate power consumption of 3622 Joules per run.2. EPYC 9554 2P: Approximate power consumption of 3141 Joules per run.

OpenBenchmarking.orgGFLOPS/s, More Is Betterlibxsmm 2-1.17-3645M N K: 32EPYC 9654 2PEPYC 9554 2P30060090012001500SE +/- 9.98, N = 15SE +/- 4.06, N = 81298.01402.21. (CXX) g++ options: -dynamic -Bstatic -static-libgcc -lgomp -lm -lrt -ldl -lquadmath -lstdc++ -pthread -fPIC -std=c++14 -O2 -fopenmp-simd -funroll-loops -ftree-vectorize -fdata-sections -ffunction-sections -fvisibility=hidden -msse4.2
OpenBenchmarking.orgGFLOPS/s Per Watt, More Is Betterlibxsmm 2-1.17-3645M N K: 32EPYC 9654 2PEPYC 9554 2P1.0762.1523.2284.3045.383.6094.782
MinAvgMaxEPYC 9654 2P94.5359.7590.3EPYC 9554 2P46.9293.2543.9OpenBenchmarking.orgWatts, Fewer Is Betterlibxsmm 2-1.17-3645CPU Power Consumption Monitor1603204806408001. EPYC 9654 2P: Approximate power consumption of 3309 Joules per run.2. EPYC 9554 2P: Approximate power consumption of 2566 Joules per run.

oneDNN

This is a test of the Intel oneDNN as an Intel-optimized library for Deep Neural Networks and making use of its built-in benchdnn functionality. The result is the total perf time reported. Intel oneDNN was formerly known as DNNL (Deep Neural Network Library) and MKL-DNN before being rebranded as part of the Intel oneAPI toolkit. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgms, Fewer Is BetteroneDNN 3.1Harness: Deconvolution Batch shapes_3d - Data Type: bf16bf16bf16 - Engine: CPUXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P0.79541.59082.38623.18163.977SE +/- 0.029518, N = 15SE +/- 0.004782, N = 9SE +/- 0.002054, N = 93.5352500.7321000.457272MIN: 2.461. (CXX) g++ options: -O3 -march=native -fopenmp -msse4.1 -fPIC -pie -ldl -lpthread
MinAvgMaxXeon Max 9480 2P, HBM Caching268426673EPYC 9654 2P93286525EPYC 9554 2P47270572OpenBenchmarking.orgWatts, Fewer Is BetteroneDNN 3.1CPU Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 3042 Joules per run.2. EPYC 9654 2P: Approximate power consumption of 2037 Joules per run.3. EPYC 9554 2P: Approximate power consumption of 1921 Joules per run.
MinAvgMaxXeon Max 9480 2P, HBM Caching190031073507OpenBenchmarking.orgMegahertz, More Is BetteroneDNN 3.1CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, HBM Caching404592897OpenBenchmarking.orgWatts, Fewer Is BetteroneDNN 3.1System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 4220 Joules per run.

System Power Consumption Monitor

OpenBenchmarking.orgWattsSystem Power Consumption MonitorPhoronix Test Suite System MonitoringXeon Max 9480 2P, HBM Caching160320480640800Min: 260 / Avg: 751.74 / Max: 907

CPU Peak Freq (Highest CPU Core Frequency) Monitor

OpenBenchmarking.orgMegahertzCPU Peak Freq (Highest CPU Core Frequency) MonitorPhoronix Test Suite System MonitoringXeon Max 9480 2P, HBM Caching6001200180024003000Min: 1100 / Avg: 3207.85 / Max: 3577

CPU Power Consumption Monitor

OpenBenchmarking.orgWattsCPU Power Consumption MonitorPhoronix Test Suite System MonitoringXeon Max 9480 2P, HBM CachingEPYC 9654 2PEPYC 9554 2P160320480640800Min: 87.33 / Avg: 548.37 / Max: 899.87Min: 31.88 / Avg: 448.81 / Max: 725.51Min: 34.39 / Avg: 413.82 / Max: 710.63

60 Results Shown

TensorFlow
libxsmm
TensorFlow
ONNX Runtime:
  bertsquad-12 - CPU - Standard:
    Inference Time Cost (ms)
    Inferences Per Second
libxsmm
ONNX Runtime:
  CaffeNet 12-int8 - CPU - Standard:
    Inference Time Cost (ms)
    Inferences Per Second
  GPT-2 - CPU - Standard:
    Inference Time Cost (ms)
    Inferences Per Second
  ResNet50 v1-12-int8 - CPU - Standard:
    Inference Time Cost (ms)
    Inferences Per Second
oneDNN:
  Recurrent Neural Network Inference - bf16bf16bf16 - CPU
  Recurrent Neural Network Training - bf16bf16bf16 - CPU
ONNX Runtime:
  fcn-resnet101-11 - CPU - Standard:
    Inference Time Cost (ms)
    Inferences Per Second
  ArcFace ResNet-100 - CPU - Standard:
    Inference Time Cost (ms)
    Inferences Per Second
  yolov4 - CPU - Standard:
    Inference Time Cost (ms)
    Inferences Per Second
TensorFlow:
  CPU - 16 - VGG-16
  CPU - 512 - GoogLeNet
OpenVINO:
  Vehicle Detection FP16 - CPU:
    ms
    FPS
  Age Gender Recognition Retail 0013 FP16-INT8 - CPU:
    ms
    FPS
TensorFlow
OpenVINO:
  Face Detection FP16-INT8 - CPU:
    ms
    FPS
oneDNN
OpenVINO:
  Face Detection FP16 - CPU:
    ms
    FPS
  Person Detection FP16 - CPU:
    ms
    FPS
  Person Vehicle Bike Detection FP16 - CPU:
    ms
    FPS
  Person Detection FP32 - CPU:
    ms
    FPS
TensorFlow
OpenVINO:
  Machine Translation EN To DE FP16 - CPU:
    ms
    FPS
  Vehicle Detection FP16-INT8 - CPU:
    ms
    FPS
  Weld Porosity Detection FP16-INT8 - CPU:
    ms
    FPS
  Age Gender Recognition Retail 0013 FP16 - CPU:
    ms
    FPS
  Weld Porosity Detection FP16 - CPU:
    ms
    FPS
TensorFlow
oneDNN
TensorFlow
oneDNN:
  IP Shapes 3D - bf16bf16bf16 - CPU
  Convolution Batch Shapes Auto - bf16bf16bf16 - CPU
libxsmm:
  64
  32
oneDNN
System Power Consumption Monitor:
  Phoronix Test Suite System Monitoring:
    Watts
    Megahertz
    Watts

EPYC 9654 2P

Processor: 2 x AMD EPYC 9654 96-Core @ 2.40GHz (192 Cores / 384 Threads), Motherboard: AMD Titanite_4G (RTI1007B BIOS), Chipset: AMD Device 14a4, Memory: 1520GB, Disk: 7682GB INTEL SSDPF2KX076TZ, Graphics: ASPEED, Monitor: VGA HDMI, Network: Broadcom NetXtreme BCM5720 PCIe

OS: Ubuntu 23.04, Kernel: 6.2.0-20-generic (x86_64), Desktop: GNOME Shell 44.0, Display Server: X Server 1.21.1.7, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1080

Kernel Notes: Transparent Huge Pages: madvise
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa101121
Python Notes: Python 3.11.2
Security Notes: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 28 June 2023 15:03 by user phoronix.

EPYC 9554 2P

Processor: 2 x AMD EPYC 9554 64-Core @ 3.10GHz (128 Cores / 256 Threads), Motherboard: AMD Titanite_4G (RTI1007B BIOS), Chipset: AMD Device 14a4, Memory: 1520GB, Disk: 7682GB INTEL SSDPF2KX076TZ, Graphics: ASPEED, Monitor: VGA HDMI, Network: Broadcom NetXtreme BCM5720 PCIe

OS: Ubuntu 23.04, Kernel: 6.2.0-20-generic (x86_64), Desktop: GNOME Shell 44.0, Display Server: X Server 1.21.1.7, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1080

Kernel Notes: Transparent Huge Pages: madvise
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa101121
Python Notes: Python 3.11.2
Security Notes: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 28 June 2023 20:58 by user phoronix.

Xeon Max 9480 2P, HBM Caching

Processor: 2 x Intel Xeon Max 9480 @ 3.50GHz (112 Cores / 224 Threads), Motherboard: Supermicro X13DEM v1.10 (1.3 BIOS), Chipset: Intel Device 1bce, Memory: 512GB, Disk: 7682GB INTEL SSDPF2KX076TZ, Graphics: ASPEED, Monitor: VE228, Network: 2 x Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb

OS: Ubuntu 23.04, Kernel: 6.2.0-20-generic (x86_64), Desktop: GNOME Shell 44.0, Display Server: X Server 1.21.1.7, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1080

Kernel Notes: Transparent Huge Pages: madvise
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c0001d1
Python Notes: Python 3.11.2
Security Notes: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 29 June 2023 09:57 by user phoronix.