Intel Xeon Max AMX HBM2e Performance Benchmark

Benchmarks for a future article on Phoronix by Michael Larabel.

Compare your own system(s) to this result file with the Phoronix Test Suite by running the command: phoronix-test-suite benchmark 2307017-NE-XEONMAXAM71
Jump To Table - Results

View

Do Not Show Noisy Results
Do Not Show Results With Incomplete Data
Do Not Show Results With Little Change/Spread
List Notable Results
Show Result Confidence Charts

Limit displaying results to tests within:

HPC - High Performance Computing 2 Tests
Machine Learning 2 Tests
Python Tests 2 Tests

Statistics

Show Overall Harmonic Mean(s)
Show Overall Geometric Mean
Show Geometric Means Per-Suite/Category
Show Wins / Losses Counts (Pie Chart)
Normalize Results
Remove Outliers Before Calculating Averages

Graph Settings

Force Line Graphs Where Applicable
Convert To Scalar Where Applicable
Disable Color Branding
Prefer Vertical Bar Graphs
No Box Plots
On Line Graphs With Missing Data, Connect The Line Gaps

Additional Graphs

Show Perf Per Core/Thread Calculation Graphs Where Applicable
Show Perf Per Clock Calculation Graphs Where Applicable

Multi-Way Comparison

Condense Multi-Option Tests Into Single Result Graphs
Condense Test Profiles With Multiple Version Results Into Single Result Graphs

Table

Show Detailed System Result Table

Run Management

Highlight
Result
Hide
Result
Result
Identifier
Performance Per
Dollar
Date
Run
  Test
  Duration
Xeon Max 9480 2P, No HBM, Max AVX512 FP16
July 01 2023
  1 Hour, 35 Minutes
Xeon Max 9480 2P, No HBM
June 30 2023
  2 Hours, 13 Minutes
Xeon Max 9480 2P, HBM Caching
June 29 2023
  2 Hours, 13 Minutes
Xeon Max 9480 2P, HBM Only
June 30 2023
  1 Hour, 35 Minutes
EPYC 9554 2P
June 28 2023
  1 Hour, 36 Minutes
EPYC 9654 2P
June 28 2023
  1 Hour, 34 Minutes
Invert Hiding All Results Option
  1 Hour, 48 Minutes

Only show results where is faster than
Only show results matching title/arguments (delimit multiple options with a comma):
Do not show results matching title/arguments (delimit multiple options with a comma):

Xeon Max 9480 2P, No HBM, Max AVX512 FP16

Kernel Notes: Transparent Huge Pages: madvise
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c0001d1
Python Notes: Python 3.11.2
Security Notes: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

Xeon Max 9480 2P, No HBM

Xeon Max 9480 2P, HBM Caching

Processor: 2 x Intel Xeon Max 9480 @ 3.50GHz (112 Cores / 224 Threads), Motherboard: Supermicro X13DEM v1.10 (1.3 BIOS), Chipset: Intel Device 1bce, Memory: 512GB, Disk: 7682GB INTEL SSDPF2KX076TZ, Graphics: ASPEED, Monitor: VE228, Network: 2 x Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb

OS: Ubuntu 23.04, Kernel: 6.2.0-20-generic (x86_64), Desktop: GNOME Shell 44.0, Display Server: X Server 1.21.1.7, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1080

Xeon Max 9480 2P, HBM Only

Changed Memory to 128GB.

EPYC 9554 2P

Processor: 2 x AMD EPYC 9554 64-Core @ 3.10GHz (128 Cores / 256 Threads), Motherboard: AMD Titanite_4G (RTI1007B BIOS), Chipset: AMD Device 14a4, Memory: 1520GB, Disk: 7682GB INTEL SSDPF2KX076TZ, Graphics: ASPEED, Monitor: VGA HDMI, Network: Broadcom NetXtreme BCM5720 PCIe

Kernel Notes: Transparent Huge Pages: madvise
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa101121
Python Notes: Python 3.11.2
Security Notes: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected

EPYC 9654 2P

Changed Processor to 2 x AMD EPYC 9654 96-Core @ 2.40GHz (192 Cores / 384 Threads).

Intel Xeon Max AMX HBM2e Performance BenchmarkProcessorMotherboardChipsetMemoryDiskGraphicsMonitorNetworkOSKernelDesktopDisplay ServerCompilerFile-SystemScreen ResolutionXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P2 x Intel Xeon Max 9480 @ 3.50GHz (112 Cores / 224 Threads)Supermicro X13DEM v1.10 (1.3 BIOS)Intel Device 1bce512GB7682GB INTEL SSDPF2KX076TZASPEEDVE2282 x Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200GbUbuntu 23.046.2.0-20-generic (x86_64)GNOME Shell 44.0X Server 1.21.1.7GCC 12.2.0ext41920x1080128GB2 x AMD EPYC 9554 64-Core @ 3.10GHz (128 Cores / 256 Threads)AMD Titanite_4G (RTI1007B BIOS)AMD Device 14a41520GBVGA HDMIBroadcom NetXtreme BCM5720 PCIe2 x AMD EPYC 9654 96-Core @ 2.40GHz (192 Cores / 384 Threads)OpenBenchmarking.orgKernel Details- Transparent Huge Pages: madviseCompiler Details- --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details- Xeon Max 9480 2P, No HBM, Max AVX512 FP16: Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c0001d1- Xeon Max 9480 2P, No HBM: Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c0001d1- Xeon Max 9480 2P, HBM Caching: Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c0001d1- Xeon Max 9480 2P, HBM Only: Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c0001d1- EPYC 9554 2P: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa101121- EPYC 9654 2P: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa101121Python Details- Python 3.11.2Security Details- Xeon Max 9480 2P, No HBM, Max AVX512 FP16: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected - Xeon Max 9480 2P, No HBM: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected - Xeon Max 9480 2P, HBM Caching: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected - Xeon Max 9480 2P, HBM Only: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected - EPYC 9554 2P: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected - EPYC 9654 2P: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected

Xeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2PLogarithmic Result OverviewPhoronix Test SuiteOpenVINOOpenVINOOpenVINOOpenVINOOpenVINOOpenVINOOpenVINOOpenVINOOpenVINOOpenVINOONNX RuntimeOpenVINOOpenVINOONNX RuntimeONNX RuntimeONNX RuntimeW.P.D.F - CPUF.D.F - CPUF.D.F - CPUP.D.F - CPUP.D.F - CPUF.D.F.I - CPUP.D.F - CPUW.P.D.F.I - CPUP.D.F - CPUW.P.D.F.I - CPUGPT-2 - CPU - StandardW.P.D.F - CPUF.D.F.I - CPUCaffeNet 12-int8 - CPU - StandardGPT-2 - CPU - StandardCaffeNet 12-int8 - CPU - Standard

Intel Xeon Max AMX HBM2e Performance Benchmarkopenvino: Face Detection FP16 - CPUopenvino: Person Detection FP16 - CPUopenvino: Person Detection FP32 - CPUopenvino: Face Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUonnx: GPT-2 - CPU - Standardonnx: CaffeNet 12-int8 - CPU - Standardonnx: GPT-2 - CPU - Standardonnx: CaffeNet 12-int8 - CPU - Standardopenvino: Face Detection FP16 - CPUopenvino: Person Detection FP16 - CPUopenvino: Person Detection FP32 - CPUopenvino: Face Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P19.8311.3111.26106.271977.0810680.95157.758543.8116.416921.837971398.322445.492449.29262.8456.6010.47115.2032.5233.79293.4916532.4031482.78155.155537.8216.508891.86415242.64857.32824.93380.606.653.45120.4033.4033.97334.8916941.9930938.50218.692562.2184.708491.78296232.01834.97819.76333.516.443.47175.7439.4738.77369.6418452.0235643.79246.700596.9084.052371.68307159.13706.83719.49302.455.923.0782.0438.2438.53155.288039.5715659.32132.721495.2407.647352.01949389.53832.02825.52205.713.978.09101.5643.6443.40191.419843.0319170.61103.184435.1949.689042.30611471.821092.971098.96250.394.869.95OpenBenchmarking.org

CPU Peak Freq (Highest CPU Core Frequency) Monitor

OpenBenchmarking.orgMegahertzCPU Peak Freq (Highest CPU Core Frequency) MonitorPhoronix Test Suite System MonitoringXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM Only6001200180024003000Min: 1800 / Avg: 3336.55 / Max: 3518Min: 1000 / Avg: 3056.85 / Max: 3576Min: 1100 / Avg: 3207.85 / Max: 3577Min: 1566 / Avg: 3308.69 / Max: 3551

CPU Power Consumption Monitor

OpenBenchmarking.orgWattsCPU Power Consumption MonitorPhoronix Test Suite System MonitoringXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P2004006008001000Min: 166.66 / Avg: 528.06 / Max: 636.62Min: 93.37 / Avg: 522.99 / Max: 664.03Min: 87.33 / Avg: 548.37 / Max: 899.87Min: 129.15 / Avg: 563.95 / Max: 1026.74Min: 34.39 / Avg: 413.82 / Max: 710.63Min: 31.88 / Avg: 448.81 / Max: 725.51

System Power Consumption Monitor

OpenBenchmarking.orgWattsSystem Power Consumption MonitorPhoronix Test Suite System MonitoringXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM Only160320480640800Min: 276 / Avg: 733.34 / Max: 937Min: 251 / Avg: 735.78 / Max: 904Min: 260 / Avg: 751.74 / Max: 907Min: 250 / Avg: 752.26 / Max: 929

OpenVINO

This is a test of the Intel OpenVINO, a toolkit around neural networks, using its built-in benchmarking support and analyzing the throughput and latency for various models. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Face Detection FP16 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P4080120160200SE +/- 0.11, N = 3SE +/- 0.29, N = 3SE +/- 0.41, N = 3SE +/- 0.12, N = 3SE +/- 0.04, N = 3SE +/- 0.25, N = 319.83115.20120.40175.7482.04101.561. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Person Detection FP16 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P1020304050SE +/- 0.04, N = 3SE +/- 0.42, N = 3SE +/- 0.24, N = 3SE +/- 0.08, N = 3SE +/- 0.23, N = 3SE +/- 0.21, N = 311.3132.5233.4039.4738.2443.641. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Person Detection FP32 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P1020304050SE +/- 0.07, N = 3SE +/- 0.25, N = 3SE +/- 0.31, N = 3SE +/- 0.42, N = 3SE +/- 0.04, N = 3SE +/- 0.20, N = 311.2633.7933.9738.7738.5343.401. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P80160240320400SE +/- 0.10, N = 3SE +/- 0.25, N = 3SE +/- 0.45, N = 3SE +/- 0.06, N = 3SE +/- 0.14, N = 3SE +/- 0.06, N = 3106.27293.49334.89369.64155.28191.411. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P4K8K12K16K20KSE +/- 2.85, N = 3SE +/- 65.25, N = 3SE +/- 13.63, N = 3SE +/- 33.75, N = 3SE +/- 1.22, N = 3SE +/- 2.32, N = 31977.0816532.4016941.9918452.028039.579843.031. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P8K16K24K32K40KSE +/- 12.56, N = 3SE +/- 37.83, N = 3SE +/- 38.32, N = 3SE +/- 37.65, N = 3SE +/- 3.60, N = 3SE +/- 4.81, N = 310680.9531482.7830938.5035643.7915659.3219170.611. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

ONNX Runtime

ONNX Runtime is developed by Microsoft and partners as a open-source, cross-platform, high performance machine learning inferencing and training accelerator. This test profile runs the ONNX Runtime with various models available from the ONNX Model Zoo. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgInferences Per Second, More Is BetterONNX Runtime 1.14Model: GPT-2 - Device: CPU - Executor: StandardXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P50100150200250SE +/- 4.54, N = 15SE +/- 4.09, N = 15SE +/- 9.23, N = 15SE +/- 3.11, N = 3SE +/- 4.25, N = 15SE +/- 0.03, N = 3157.76155.16218.69246.70132.72103.181. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt
OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: GPT-2 - Device: CPU - Executor: StandardXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P3691215SE +/- 0.19970, N = 15SE +/- 0.18098, N = 15SE +/- 0.23765, N = 15SE +/- 0.05143, N = 3SE +/- 0.25707, N = 15SE +/- 0.00311, N = 36.416926.508894.708494.052377.647359.689041. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt
OpenBenchmarking.orgMegahertz, More Is BetterONNX Runtime 1.14CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, No HBM, Max AVX512 FP166001200180024003000Min: 3500 / Avg: 3501.18 / Max: 3518
OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14System Power Consumption MonitorXeon Max 9480 2P, No HBM, Max AVX512 FP16130260390520650Min: 386 / Avg: 718.32 / Max: 7491. Xeon Max 9480 2P, No HBM, Max AVX512 FP16: Approximate power consumption of 68432 Joules per run.

OpenBenchmarking.orgInferences Per Second, More Is BetterONNX Runtime 1.14Model: CaffeNet 12-int8 - Device: CPU - Executor: StandardXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P130260390520650SE +/- 3.29, N = 3SE +/- 7.97, N = 15SE +/- 8.12, N = 15SE +/- 10.95, N = 15SE +/- 5.94, N = 4SE +/- 7.41, N = 15543.81537.82562.22596.91495.24435.191. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt
OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: CaffeNet 12-int8 - Device: CPU - Executor: StandardXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P0.51891.03781.55672.07562.5945SE +/- 0.01109, N = 3SE +/- 0.02958, N = 15SE +/- 0.02850, N = 15SE +/- 0.03485, N = 15SE +/- 0.02472, N = 4SE +/- 0.03734, N = 151.837971.864151.782961.683072.019492.306111. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt
OpenBenchmarking.orgMegahertz, More Is BetterONNX Runtime 1.14CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, No HBM, Max AVX512 FP1680016002400320040003500
MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16416765857OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, No HBM, Max AVX512 FP16: Approximate power consumption of 72126 Joules per run.

OpenVINO

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16350035003512Xeon Max 9480 2P, No HBM110016543503Xeon Max 9480 2P, HBM Caching120017403505Xeon Max 9480 2P, HBM Only200034933508OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16220034813515Xeon Max 9480 2P, No HBM180033133518Xeon Max 9480 2P, HBM Caching190030233512Xeon Max 9480 2P, HBM Only350035003517OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16350035003511Xeon Max 9480 2P, No HBM208733963518Xeon Max 9480 2P, HBM Caching190030743506Xeon Max 9480 2P, HBM Only264634963508OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16180024923510Xeon Max 9480 2P, No HBM150021953515Xeon Max 9480 2P, HBM Caching170023703505Xeon Max 9480 2P, HBM Only190033593505OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16350035003509Xeon Max 9480 2P, No HBM160018463500Xeon Max 9480 2P, HBM Caching170019003500Xeon Max 9480 2P, HBM Only190026183506OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16180020233500Xeon Max 9480 2P, No HBM160019343502Xeon Max 9480 2P, HBM Caching150019023504Xeon Max 9480 2P, HBM Only180029323508OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Face Detection FP16 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P30060090012001500SE +/- 8.00, N = 3SE +/- 0.61, N = 3SE +/- 0.80, N = 3SE +/- 0.10, N = 3SE +/- 0.19, N = 3SE +/- 0.93, N = 31398.32242.64232.01159.13389.53471.82MIN: 943.82 / MAX: 1957.07MIN: 380.82 / MAX: 432.57MIN: 431.51 / MAX: 569.31. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF
MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16259561637Xeon Max 9480 2P, No HBM163519586Xeon Max 9480 2P, HBM Caching136533628Xeon Max 9480 2P, HBM Only256574649EPYC 9554 2P49638711EPYC 9654 2P58646726OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, No HBM, Max AVX512 FP16: Approximate power consumption of 39802 Joules per run.2. Xeon Max 9480 2P, No HBM: Approximate power consumption of 38211 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 39944 Joules per run.4. Xeon Max 9480 2P, HBM Only: Approximate power consumption of 40360 Joules per run.5. EPYC 9554 2P: Approximate power consumption of 43188 Joules per run.6. EPYC 9654 2P: Approximate power consumption of 44817 Joules per run.
MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16350035003512Xeon Max 9480 2P, No HBM110016543503Xeon Max 9480 2P, HBM Caching120017403505Xeon Max 9480 2P, HBM Only200034933508OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16399769937Xeon Max 9480 2P, No HBM274734840Xeon Max 9480 2P, HBM Caching281724839Xeon Max 9480 2P, HBM Only362757833OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, No HBM, Max AVX512 FP16: Approximate power consumption of 54589 Joules per run.2. Xeon Max 9480 2P, No HBM: Approximate power consumption of 54036 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 54275 Joules per run.4. Xeon Max 9480 2P, HBM Only: Approximate power consumption of 53253 Joules per run.

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Person Detection FP16 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P5001000150020002500SE +/- 9.42, N = 3SE +/- 10.44, N = 3SE +/- 6.17, N = 3SE +/- 1.42, N = 3SE +/- 5.30, N = 3SE +/- 5.11, N = 32445.49857.32834.97706.83832.021092.97MIN: 1611.66 / MAX: 3114.74MIN: 481.73 / MAX: 1422.59MIN: 558.93 / MAX: 2258.39MIN: 455.92 / MAX: 1885.53MIN: 743.95 / MAX: 1258.17MIN: 825.28 / MAX: 1814.821. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF
MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16267546628Xeon Max 9480 2P, No HBM205515616Xeon Max 9480 2P, HBM Caching204538618Xeon Max 9480 2P, HBM Only252563639EPYC 9554 2P49596683EPYC 9654 2P60608699OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, No HBM, Max AVX512 FP16: Approximate power consumption of 43880 Joules per run.2. Xeon Max 9480 2P, No HBM: Approximate power consumption of 36054 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 38946 Joules per run.4. Xeon Max 9480 2P, HBM Only: Approximate power consumption of 39997 Joules per run.5. EPYC 9554 2P: Approximate power consumption of 41100 Joules per run.6. EPYC 9654 2P: Approximate power consumption of 42760 Joules per run.
MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16220034813515Xeon Max 9480 2P, No HBM180033133518Xeon Max 9480 2P, HBM Caching190030233512Xeon Max 9480 2P, HBM Only350035003517OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16408757822Xeon Max 9480 2P, No HBM295737874Xeon Max 9480 2P, HBM Caching298736817Xeon Max 9480 2P, HBM Only360745835OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, No HBM, Max AVX512 FP16: Approximate power consumption of 60783 Joules per run.2. Xeon Max 9480 2P, No HBM: Approximate power consumption of 52049 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 53264 Joules per run.4. Xeon Max 9480 2P, HBM Only: Approximate power consumption of 52914 Joules per run.

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Person Detection FP32 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P5001000150020002500SE +/- 11.98, N = 3SE +/- 6.05, N = 3SE +/- 7.20, N = 3SE +/- 7.70, N = 3SE +/- 1.07, N = 3SE +/- 5.33, N = 32449.29824.93819.76719.49825.521098.96MIN: 1558.62 / MAX: 3306.73MIN: 445.3 / MAX: 1462.45MIN: 516.88 / MAX: 2598.39MIN: 473.54 / MAX: 2264.09MIN: 747.05 / MAX: 1238.39MIN: 825.62 / MAX: 1835.761. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF
MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16241553629Xeon Max 9480 2P, No HBM245531618Xeon Max 9480 2P, HBM Caching228547624Xeon Max 9480 2P, HBM Only223567643EPYC 9554 2P49600684EPYC 9654 2P60606703OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, No HBM, Max AVX512 FP16: Approximate power consumption of 39605 Joules per run.2. Xeon Max 9480 2P, No HBM: Approximate power consumption of 36803 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 38131 Joules per run.4. Xeon Max 9480 2P, HBM Only: Approximate power consumption of 39091 Joules per run.5. EPYC 9554 2P: Approximate power consumption of 41612 Joules per run.6. EPYC 9654 2P: Approximate power consumption of 42612 Joules per run.
MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16350035003511Xeon Max 9480 2P, No HBM208733963518Xeon Max 9480 2P, HBM Caching190030743506Xeon Max 9480 2P, HBM Only264634963508OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16413768860Xeon Max 9480 2P, No HBM378758872Xeon Max 9480 2P, HBM Caching369748857Xeon Max 9480 2P, HBM Only373752832OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, No HBM, Max AVX512 FP16: Approximate power consumption of 55021 Joules per run.2. Xeon Max 9480 2P, No HBM: Approximate power consumption of 52581 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 52353 Joules per run.4. Xeon Max 9480 2P, HBM Only: Approximate power consumption of 52108 Joules per run.

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P80160240320400SE +/- 0.33, N = 3SE +/- 0.34, N = 3SE +/- 0.44, N = 3SE +/- 0.04, N = 3SE +/- 0.22, N = 3SE +/- 0.04, N = 3262.84380.60333.51302.45205.71250.39MIN: 192.97 / MAX: 452.59MIN: 286.27 / MAX: 602.06MIN: 285.32 / MAX: 516.28MIN: 217.26 / MAX: 450.48MIN: 202.38 / MAX: 252.14MIN: 222.36 / MAX: 303.721. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF
MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16184553616Xeon Max 9480 2P, No HBM157500632Xeon Max 9480 2P, HBM Caching168531616Xeon Max 9480 2P, HBM Only249561682EPYC 9554 2P50606675EPYC 9654 2P59613687OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, No HBM, Max AVX512 FP16: Approximate power consumption of 37939 Joules per run.2. Xeon Max 9480 2P, No HBM: Approximate power consumption of 43309 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 46887 Joules per run.4. Xeon Max 9480 2P, HBM Only: Approximate power consumption of 46730 Joules per run.5. EPYC 9554 2P: Approximate power consumption of 41215 Joules per run.6. EPYC 9654 2P: Approximate power consumption of 42688 Joules per run.
MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16180024923510Xeon Max 9480 2P, No HBM150021953515Xeon Max 9480 2P, HBM Caching170023703505Xeon Max 9480 2P, HBM Only190033593505OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16307762847Xeon Max 9480 2P, No HBM256717841Xeon Max 9480 2P, HBM Caching270722818Xeon Max 9480 2P, HBM Only365742888OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, No HBM, Max AVX512 FP16: Approximate power consumption of 52850 Joules per run.2. Xeon Max 9480 2P, No HBM: Approximate power consumption of 62122 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 64059 Joules per run.4. Xeon Max 9480 2P, HBM Only: Approximate power consumption of 61794 Joules per run.

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P1326395265SE +/- 0.08, N = 3SE +/- 0.03, N = 3SE +/- 0.01, N = 3SE +/- 0.01, N = 3SE +/- 0.00, N = 3SE +/- 0.00, N = 356.606.656.445.923.974.86MIN: 44.94 / MAX: 112.121. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF
MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16268564605Xeon Max 9480 2P, No HBM165538595Xeon Max 9480 2P, HBM Caching163557629Xeon Max 9480 2P, HBM Only235582661EPYC 9554 2P48648698EPYC 9654 2P57655705OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, No HBM, Max AVX512 FP16: Approximate power consumption of 36827 Joules per run.2. Xeon Max 9480 2P, No HBM: Approximate power consumption of 35883 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 37337 Joules per run.4. Xeon Max 9480 2P, HBM Only: Approximate power consumption of 38007 Joules per run.5. EPYC 9554 2P: Approximate power consumption of 41467 Joules per run.6. EPYC 9654 2P: Approximate power consumption of 42138 Joules per run.
MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16350035003509Xeon Max 9480 2P, No HBM160018463500Xeon Max 9480 2P, HBM Caching170019003500Xeon Max 9480 2P, HBM Only190026183506OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16420774811Xeon Max 9480 2P, No HBM268759840Xeon Max 9480 2P, HBM Caching279756840Xeon Max 9480 2P, HBM Only342768875OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, No HBM, Max AVX512 FP16: Approximate power consumption of 50568 Joules per run.2. Xeon Max 9480 2P, No HBM: Approximate power consumption of 50606 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 50630 Joules per run.4. Xeon Max 9480 2P, HBM Only: Approximate power consumption of 50410 Joules per run.

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P3691215SE +/- 0.01, N = 3SE +/- 0.00, N = 3SE +/- 0.00, N = 3SE +/- 0.01, N = 3SE +/- 0.00, N = 3SE +/- 0.00, N = 310.473.453.473.078.099.95MIN: 7.24 / MAX: 56.73MIN: 2.47 / MAX: 46.27MIN: 2.48 / MAX: 56.57MIN: 2.48 / MAX: 68.92MIN: 7.89 / MAX: 45.39MIN: 8.36 / MAX: 40.391. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF
MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16168556603Xeon Max 9480 2P, No HBM161541594Xeon Max 9480 2P, HBM Caching137552614Xeon Max 9480 2P, HBM Only256583675EPYC 9554 2P49623675EPYC 9654 2P59632685OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, No HBM, Max AVX512 FP16: Approximate power consumption of 36481 Joules per run.2. Xeon Max 9480 2P, No HBM: Approximate power consumption of 36451 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 37176 Joules per run.4. Xeon Max 9480 2P, HBM Only: Approximate power consumption of 38286 Joules per run.5. EPYC 9554 2P: Approximate power consumption of 40518 Joules per run.6. EPYC 9654 2P: Approximate power consumption of 41474 Joules per run.
MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16180020233500Xeon Max 9480 2P, No HBM160019343502Xeon Max 9480 2P, HBM Caching150019023504Xeon Max 9480 2P, HBM Only180029323508OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16276765811Xeon Max 9480 2P, No HBM284753808Xeon Max 9480 2P, HBM Caching275752830Xeon Max 9480 2P, HBM Only370768854OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor20040060080010001. Xeon Max 9480 2P, No HBM, Max AVX512 FP16: Approximate power consumption of 50222 Joules per run.2. Xeon Max 9480 2P, No HBM: Approximate power consumption of 50717 Joules per run.3. Xeon Max 9480 2P, HBM Caching: Approximate power consumption of 51165 Joules per run.4. Xeon Max 9480 2P, HBM Only: Approximate power consumption of 50458 Joules per run.

Xeon Max 9480 2P, No HBM, Max AVX512 FP16

Kernel Notes: Transparent Huge Pages: madvise
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c0001d1
Python Notes: Python 3.11.2
Security Notes: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 1 July 2023 13:22 by user phoronix.

Xeon Max 9480 2P, No HBM

Kernel Notes: Transparent Huge Pages: madvise
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c0001d1
Python Notes: Python 3.11.2
Security Notes: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 30 June 2023 05:43 by user phoronix.

Xeon Max 9480 2P, HBM Caching

Processor: 2 x Intel Xeon Max 9480 @ 3.50GHz (112 Cores / 224 Threads), Motherboard: Supermicro X13DEM v1.10 (1.3 BIOS), Chipset: Intel Device 1bce, Memory: 512GB, Disk: 7682GB INTEL SSDPF2KX076TZ, Graphics: ASPEED, Monitor: VE228, Network: 2 x Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb

OS: Ubuntu 23.04, Kernel: 6.2.0-20-generic (x86_64), Desktop: GNOME Shell 44.0, Display Server: X Server 1.21.1.7, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1080

Kernel Notes: Transparent Huge Pages: madvise
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c0001d1
Python Notes: Python 3.11.2
Security Notes: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 29 June 2023 09:57 by user phoronix.

Xeon Max 9480 2P, HBM Only

Processor: 2 x Intel Xeon Max 9480 @ 3.50GHz (112 Cores / 224 Threads), Motherboard: Supermicro X13DEM v1.10 (1.3 BIOS), Chipset: Intel Device 1bce, Memory: 128GB, Disk: 7682GB INTEL SSDPF2KX076TZ, Graphics: ASPEED, Monitor: VE228, Network: 2 x Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb

OS: Ubuntu 23.04, Kernel: 6.2.0-20-generic (x86_64), Desktop: GNOME Shell 44.0, Display Server: X Server 1.21.1.7, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1080

Kernel Notes: Transparent Huge Pages: madvise
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c0001d1
Python Notes: Python 3.11.2
Security Notes: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 30 June 2023 11:10 by user phoronix.

EPYC 9554 2P

Processor: 2 x AMD EPYC 9554 64-Core @ 3.10GHz (128 Cores / 256 Threads), Motherboard: AMD Titanite_4G (RTI1007B BIOS), Chipset: AMD Device 14a4, Memory: 1520GB, Disk: 7682GB INTEL SSDPF2KX076TZ, Graphics: ASPEED, Monitor: VGA HDMI, Network: Broadcom NetXtreme BCM5720 PCIe

OS: Ubuntu 23.04, Kernel: 6.2.0-20-generic (x86_64), Desktop: GNOME Shell 44.0, Display Server: X Server 1.21.1.7, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1080

Kernel Notes: Transparent Huge Pages: madvise
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa101121
Python Notes: Python 3.11.2
Security Notes: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 28 June 2023 20:58 by user phoronix.

EPYC 9654 2P

Processor: 2 x AMD EPYC 9654 96-Core @ 2.40GHz (192 Cores / 384 Threads), Motherboard: AMD Titanite_4G (RTI1007B BIOS), Chipset: AMD Device 14a4, Memory: 1520GB, Disk: 7682GB INTEL SSDPF2KX076TZ, Graphics: ASPEED, Monitor: VGA HDMI, Network: Broadcom NetXtreme BCM5720 PCIe

OS: Ubuntu 23.04, Kernel: 6.2.0-20-generic (x86_64), Desktop: GNOME Shell 44.0, Display Server: X Server 1.21.1.7, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1080

Kernel Notes: Transparent Huge Pages: madvise
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa101121
Python Notes: Python 3.11.2
Security Notes: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 28 June 2023 15:03 by user phoronix.