Intel Xeon Max AMX HBM2e Performance Benchmark

Benchmarks for a future article on Phoronix by Michael Larabel.

HTML result view exported from: https://openbenchmarking.org/result/2307017-NE-XEONMAXAM71.

Intel Xeon Max AMX HBM2e Performance BenchmarkProcessorMotherboardChipsetMemoryDiskGraphicsMonitorNetworkOSKernelDesktopDisplay ServerCompilerFile-SystemScreen ResolutionXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P2 x Intel Xeon Max 9480 @ 3.50GHz (112 Cores / 224 Threads)Supermicro X13DEM v1.10 (1.3 BIOS)Intel Device 1bce512GB7682GB INTEL SSDPF2KX076TZASPEEDVE2282 x Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200GbUbuntu 23.046.2.0-20-generic (x86_64)GNOME Shell 44.0X Server 1.21.1.7GCC 12.2.0ext41920x1080128GB2 x AMD EPYC 9554 64-Core @ 3.10GHz (128 Cores / 256 Threads)AMD Titanite_4G (RTI1007B BIOS)AMD Device 14a41520GBVGA HDMIBroadcom NetXtreme BCM5720 PCIe2 x AMD EPYC 9654 96-Core @ 2.40GHz (192 Cores / 384 Threads)OpenBenchmarking.orgKernel Details- Transparent Huge Pages: madviseCompiler Details- --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details- Xeon Max 9480 2P, No HBM, Max AVX512 FP16: Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c0001d1- Xeon Max 9480 2P, No HBM: Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c0001d1- Xeon Max 9480 2P, HBM Caching: Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c0001d1- Xeon Max 9480 2P, HBM Only: Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c0001d1- EPYC 9554 2P: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa101121- EPYC 9654 2P: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa101121Python Details- Python 3.11.2Security Details- Xeon Max 9480 2P, No HBM, Max AVX512 FP16: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected - Xeon Max 9480 2P, No HBM: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected - Xeon Max 9480 2P, HBM Caching: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected - Xeon Max 9480 2P, HBM Only: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected - EPYC 9554 2P: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected - EPYC 9654 2P: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected

Intel Xeon Max AMX HBM2e Performance Benchmarkopenvino: Face Detection FP16 - CPUopenvino: Face Detection FP16 - CPUopenvino: Person Detection FP16 - CPUopenvino: Person Detection FP16 - CPUopenvino: Person Detection FP32 - CPUopenvino: Person Detection FP32 - CPUopenvino: Face Detection FP16-INT8 - CPUopenvino: Face Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16 - CPUopenvino: Weld Porosity Detection FP16 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUonnx: GPT-2 - CPU - Standardonnx: GPT-2 - CPU - Standardonnx: CaffeNet 12-int8 - CPU - Standardonnx: CaffeNet 12-int8 - CPU - StandardXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P19.831398.3211.312445.4911.262449.29106.27262.841977.0856.6010680.9510.47157.7586.41692543.8111.83797115.20242.6432.52857.3233.79824.93293.49380.6016532.406.6531482.783.45155.1556.50889537.8211.86415120.40232.0133.40834.9733.97819.76334.89333.5116941.996.4430938.503.47218.6924.70849562.2181.78296175.74159.1339.47706.8338.77719.49369.64302.4518452.025.9235643.793.07246.7004.05237596.9081.6830782.04389.5338.24832.0238.53825.52155.28205.718039.573.9715659.328.09132.7217.64735495.2402.01949101.56471.8243.641092.9743.401098.96191.41250.399843.034.8619170.619.95103.1849.68904435.1942.30611OpenBenchmarking.org

OpenVINO

Model: Face Detection FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Face Detection FP16 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P4080120160200SE +/- 0.11, N = 3SE +/- 0.29, N = 3SE +/- 0.41, N = 3SE +/- 0.12, N = 3SE +/- 0.04, N = 3SE +/- 0.25, N = 319.83115.20120.40175.7482.04101.561. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

Model: Face Detection FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Face Detection FP16 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P30060090012001500SE +/- 8.00, N = 3SE +/- 0.61, N = 3SE +/- 0.80, N = 3SE +/- 0.10, N = 3SE +/- 0.19, N = 3SE +/- 0.93, N = 31398.32242.64232.01159.13389.53471.82MIN: 943.82 / MAX: 1957.07MIN: 380.82 / MAX: 432.57MIN: 431.51 / MAX: 569.31. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16259561637Xeon Max 9480 2P, No HBM163519586Xeon Max 9480 2P, HBM Caching136533628Xeon Max 9480 2P, HBM Only256574649EPYC 9554 2P49638711EPYC 9654 2P58646726OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

Model: Person Detection FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Person Detection FP16 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P1020304050SE +/- 0.04, N = 3SE +/- 0.42, N = 3SE +/- 0.24, N = 3SE +/- 0.08, N = 3SE +/- 0.23, N = 3SE +/- 0.21, N = 311.3132.5233.4039.4738.2443.641. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

Model: Person Detection FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Person Detection FP16 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P5001000150020002500SE +/- 9.42, N = 3SE +/- 10.44, N = 3SE +/- 6.17, N = 3SE +/- 1.42, N = 3SE +/- 5.30, N = 3SE +/- 5.11, N = 32445.49857.32834.97706.83832.021092.97MIN: 1611.66 / MAX: 3114.74MIN: 481.73 / MAX: 1422.59MIN: 558.93 / MAX: 2258.39MIN: 455.92 / MAX: 1885.53MIN: 743.95 / MAX: 1258.17MIN: 825.28 / MAX: 1814.821. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16267546628Xeon Max 9480 2P, No HBM205515616Xeon Max 9480 2P, HBM Caching204538618Xeon Max 9480 2P, HBM Only252563639EPYC 9554 2P49596683EPYC 9654 2P60608699OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

Model: Person Detection FP32 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Person Detection FP32 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P1020304050SE +/- 0.07, N = 3SE +/- 0.25, N = 3SE +/- 0.31, N = 3SE +/- 0.42, N = 3SE +/- 0.04, N = 3SE +/- 0.20, N = 311.2633.7933.9738.7738.5343.401. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

Model: Person Detection FP32 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Person Detection FP32 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P5001000150020002500SE +/- 11.98, N = 3SE +/- 6.05, N = 3SE +/- 7.20, N = 3SE +/- 7.70, N = 3SE +/- 1.07, N = 3SE +/- 5.33, N = 32449.29824.93819.76719.49825.521098.96MIN: 1558.62 / MAX: 3306.73MIN: 445.3 / MAX: 1462.45MIN: 516.88 / MAX: 2598.39MIN: 473.54 / MAX: 2264.09MIN: 747.05 / MAX: 1238.39MIN: 825.62 / MAX: 1835.761. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16241553629Xeon Max 9480 2P, No HBM245531618Xeon Max 9480 2P, HBM Caching228547624Xeon Max 9480 2P, HBM Only223567643EPYC 9554 2P49600684EPYC 9654 2P60606703OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

Model: Face Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P80160240320400SE +/- 0.10, N = 3SE +/- 0.25, N = 3SE +/- 0.45, N = 3SE +/- 0.06, N = 3SE +/- 0.14, N = 3SE +/- 0.06, N = 3106.27293.49334.89369.64155.28191.411. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

Model: Face Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P80160240320400SE +/- 0.33, N = 3SE +/- 0.34, N = 3SE +/- 0.44, N = 3SE +/- 0.04, N = 3SE +/- 0.22, N = 3SE +/- 0.04, N = 3262.84380.60333.51302.45205.71250.39MIN: 192.97 / MAX: 452.59MIN: 286.27 / MAX: 602.06MIN: 285.32 / MAX: 516.28MIN: 217.26 / MAX: 450.48MIN: 202.38 / MAX: 252.14MIN: 222.36 / MAX: 303.721. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16184553616Xeon Max 9480 2P, No HBM157500632Xeon Max 9480 2P, HBM Caching168531616Xeon Max 9480 2P, HBM Only249561682EPYC 9554 2P50606675EPYC 9654 2P59613687OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

Model: Weld Porosity Detection FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P4K8K12K16K20KSE +/- 2.85, N = 3SE +/- 65.25, N = 3SE +/- 13.63, N = 3SE +/- 33.75, N = 3SE +/- 1.22, N = 3SE +/- 2.32, N = 31977.0816532.4016941.9918452.028039.579843.031. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

Model: Weld Porosity Detection FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P1326395265SE +/- 0.08, N = 3SE +/- 0.03, N = 3SE +/- 0.01, N = 3SE +/- 0.01, N = 3SE +/- 0.00, N = 3SE +/- 0.00, N = 356.606.656.445.923.974.86MIN: 44.94 / MAX: 112.121. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16268564605Xeon Max 9480 2P, No HBM165538595Xeon Max 9480 2P, HBM Caching163557629Xeon Max 9480 2P, HBM Only235582661EPYC 9554 2P48648698EPYC 9654 2P57655705OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

Model: Weld Porosity Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P8K16K24K32K40KSE +/- 12.56, N = 3SE +/- 37.83, N = 3SE +/- 38.32, N = 3SE +/- 37.65, N = 3SE +/- 3.60, N = 3SE +/- 4.81, N = 310680.9531482.7830938.5035643.7915659.3219170.611. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

Model: Weld Porosity Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P3691215SE +/- 0.01, N = 3SE +/- 0.00, N = 3SE +/- 0.00, N = 3SE +/- 0.01, N = 3SE +/- 0.00, N = 3SE +/- 0.00, N = 310.473.453.473.078.099.95MIN: 7.24 / MAX: 56.73MIN: 2.47 / MAX: 46.27MIN: 2.48 / MAX: 56.57MIN: 2.48 / MAX: 68.92MIN: 7.89 / MAX: 45.39MIN: 8.36 / MAX: 40.391. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16168556603Xeon Max 9480 2P, No HBM161541594Xeon Max 9480 2P, HBM Caching137552614Xeon Max 9480 2P, HBM Only256583675EPYC 9554 2P49623675EPYC 9654 2P59632685OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

ONNX Runtime

Model: GPT-2 - Device: CPU - Executor: Standard

OpenBenchmarking.orgInferences Per Second, More Is BetterONNX Runtime 1.14Model: GPT-2 - Device: CPU - Executor: StandardXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P50100150200250SE +/- 4.54, N = 15SE +/- 4.09, N = 15SE +/- 9.23, N = 15SE +/- 3.11, N = 3SE +/- 4.25, N = 15SE +/- 0.03, N = 3157.76155.16218.69246.70132.72103.181. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

ONNX Runtime

Model: GPT-2 - Device: CPU - Executor: Standard

OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: GPT-2 - Device: CPU - Executor: StandardXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P3691215SE +/- 0.19970, N = 15SE +/- 0.18098, N = 15SE +/- 0.23765, N = 15SE +/- 0.05143, N = 3SE +/- 0.25707, N = 15SE +/- 0.00311, N = 36.416926.508894.708494.052377.647359.689041. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

ONNX Runtime

Model: CaffeNet 12-int8 - Device: CPU - Executor: Standard

OpenBenchmarking.orgInferences Per Second, More Is BetterONNX Runtime 1.14Model: CaffeNet 12-int8 - Device: CPU - Executor: StandardXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P130260390520650SE +/- 3.29, N = 3SE +/- 7.97, N = 15SE +/- 8.12, N = 15SE +/- 10.95, N = 15SE +/- 5.94, N = 4SE +/- 7.41, N = 15543.81537.82562.22596.91495.24435.191. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

ONNX Runtime

Model: CaffeNet 12-int8 - Device: CPU - Executor: Standard

OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: CaffeNet 12-int8 - Device: CPU - Executor: StandardXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P0.51891.03781.55672.07562.5945SE +/- 0.01109, N = 3SE +/- 0.02958, N = 15SE +/- 0.02850, N = 15SE +/- 0.03485, N = 15SE +/- 0.02472, N = 4SE +/- 0.03734, N = 151.837971.864151.782961.683072.019492.306111. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

CPU Power Consumption Monitor

Phoronix Test Suite System Monitoring

OpenBenchmarking.orgWattsCPU Power Consumption MonitorPhoronix Test Suite System MonitoringXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P2004006008001000Min: 166.66 / Avg: 528.06 / Max: 636.62Min: 93.37 / Avg: 522.99 / Max: 664.03Min: 87.33 / Avg: 548.37 / Max: 899.87Min: 129.15 / Avg: 563.95 / Max: 1026.74Min: 34.39 / Avg: 413.82 / Max: 710.63Min: 31.88 / Avg: 448.81 / Max: 725.51

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16350035003512Xeon Max 9480 2P, No HBM110016543503Xeon Max 9480 2P, HBM Caching120017403505Xeon Max 9480 2P, HBM Only200034933508OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

System Power Consumption Monitor

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16399769937Xeon Max 9480 2P, No HBM274734840Xeon Max 9480 2P, HBM Caching281724839Xeon Max 9480 2P, HBM Only362757833OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor2004006008001000

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16220034813515Xeon Max 9480 2P, No HBM180033133518Xeon Max 9480 2P, HBM Caching190030233512Xeon Max 9480 2P, HBM Only350035003517OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

System Power Consumption Monitor

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16408757822Xeon Max 9480 2P, No HBM295737874Xeon Max 9480 2P, HBM Caching298736817Xeon Max 9480 2P, HBM Only360745835OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor2004006008001000

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16350035003511Xeon Max 9480 2P, No HBM208733963518Xeon Max 9480 2P, HBM Caching190030743506Xeon Max 9480 2P, HBM Only264634963508OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

System Power Consumption Monitor

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16413768860Xeon Max 9480 2P, No HBM378758872Xeon Max 9480 2P, HBM Caching369748857Xeon Max 9480 2P, HBM Only373752832OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor2004006008001000

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16180024923510Xeon Max 9480 2P, No HBM150021953515Xeon Max 9480 2P, HBM Caching170023703505Xeon Max 9480 2P, HBM Only190033593505OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

System Power Consumption Monitor

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16307762847Xeon Max 9480 2P, No HBM256717841Xeon Max 9480 2P, HBM Caching270722818Xeon Max 9480 2P, HBM Only365742888OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor2004006008001000

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16350035003509Xeon Max 9480 2P, No HBM160018463500Xeon Max 9480 2P, HBM Caching170019003500Xeon Max 9480 2P, HBM Only190026183506OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

System Power Consumption Monitor

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16420774811Xeon Max 9480 2P, No HBM268759840Xeon Max 9480 2P, HBM Caching279756840Xeon Max 9480 2P, HBM Only342768875OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor2004006008001000

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16180020233500Xeon Max 9480 2P, No HBM160019343502Xeon Max 9480 2P, HBM Caching150019023504Xeon Max 9480 2P, HBM Only180029323508OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

System Power Consumption Monitor

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16276765811Xeon Max 9480 2P, No HBM284753808Xeon Max 9480 2P, HBM Caching275752830Xeon Max 9480 2P, HBM Only370768854OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor2004006008001000

CPU Peak Freq (Highest CPU Core Frequency) Monitor

Phoronix Test Suite System Monitoring

OpenBenchmarking.orgMegahertzCPU Peak Freq (Highest CPU Core Frequency) MonitorPhoronix Test Suite System MonitoringXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM Only6001200180024003000Min: 1800 / Avg: 3336.55 / Max: 3518Min: 1000 / Avg: 3056.85 / Max: 3576Min: 1100 / Avg: 3207.85 / Max: 3577Min: 1566 / Avg: 3308.69 / Max: 3551

System Power Consumption Monitor

Phoronix Test Suite System Monitoring

OpenBenchmarking.orgWattsSystem Power Consumption MonitorPhoronix Test Suite System MonitoringXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM Only160320480640800Min: 276 / Avg: 733.34 / Max: 937Min: 251 / Avg: 735.78 / Max: 904Min: 260 / Avg: 751.74 / Max: 907Min: 250 / Avg: 752.26 / Max: 929

ONNX Runtime

CPU Peak Freq (Highest CPU Core Frequency) Monitor

OpenBenchmarking.orgMegahertz, More Is BetterONNX Runtime 1.14CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, No HBM, Max AVX512 FP166001200180024003000Min: 3500 / Avg: 3501.18 / Max: 3518

ONNX Runtime

System Power Consumption Monitor

OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14System Power Consumption MonitorXeon Max 9480 2P, No HBM, Max AVX512 FP16130260390520650Min: 386 / Avg: 718.32 / Max: 749

ONNX Runtime

CPU Peak Freq (Highest CPU Core Frequency) Monitor

OpenBenchmarking.orgMegahertz, More Is BetterONNX Runtime 1.14CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, No HBM, Max AVX512 FP1680016002400320040003500

ONNX Runtime

System Power Consumption Monitor

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16416765857OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14System Power Consumption Monitor2004006008001000

Phoronix Test Suite v10.8.5