Intel Xeon Max AMX HBM2e Performance Benchmark

Benchmarks for a future article on Phoronix by Michael Larabel.

HTML result view exported from: https://openbenchmarking.org/result/2307017-NE-XEONMAXAM71&sro&gru.

Intel Xeon Max AMX HBM2e Performance BenchmarkProcessorMotherboardChipsetMemoryDiskGraphicsMonitorNetworkOSKernelDesktopDisplay ServerCompilerFile-SystemScreen ResolutionXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P2 x Intel Xeon Max 9480 @ 3.50GHz (112 Cores / 224 Threads)Supermicro X13DEM v1.10 (1.3 BIOS)Intel Device 1bce512GB7682GB INTEL SSDPF2KX076TZASPEEDVE2282 x Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200GbUbuntu 23.046.2.0-20-generic (x86_64)GNOME Shell 44.0X Server 1.21.1.7GCC 12.2.0ext41920x1080128GB2 x AMD EPYC 9554 64-Core @ 3.10GHz (128 Cores / 256 Threads)AMD Titanite_4G (RTI1007B BIOS)AMD Device 14a41520GBVGA HDMIBroadcom NetXtreme BCM5720 PCIe2 x AMD EPYC 9654 96-Core @ 2.40GHz (192 Cores / 384 Threads)OpenBenchmarking.orgKernel Details- Transparent Huge Pages: madviseCompiler Details- --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-Pa930Z/gcc-12-12.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details- Xeon Max 9480 2P, No HBM, Max AVX512 FP16: Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c0001d1- Xeon Max 9480 2P, No HBM: Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c0001d1- Xeon Max 9480 2P, HBM Caching: Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c0001d1- Xeon Max 9480 2P, HBM Only: Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c0001d1- EPYC 9554 2P: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa101121- EPYC 9654 2P: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa101121Python Details- Python 3.11.2Security Details- Xeon Max 9480 2P, No HBM, Max AVX512 FP16: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected - Xeon Max 9480 2P, No HBM: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected - Xeon Max 9480 2P, HBM Caching: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected - Xeon Max 9480 2P, HBM Only: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected - EPYC 9554 2P: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected - EPYC 9654 2P: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected

Intel Xeon Max AMX HBM2e Performance Benchmarkopenvino: Face Detection FP16 - CPUopenvino: Person Detection FP16 - CPUopenvino: Person Detection FP32 - CPUopenvino: Face Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUonnx: GPT-2 - CPU - Standardonnx: CaffeNet 12-int8 - CPU - Standardonnx: GPT-2 - CPU - Standardonnx: CaffeNet 12-int8 - CPU - Standardopenvino: Face Detection FP16 - CPUopenvino: Person Detection FP16 - CPUopenvino: Person Detection FP32 - CPUopenvino: Face Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUXeon Max 9480 2P, No HBM, Max AVX512 FP16Xeon Max 9480 2P, No HBMXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyEPYC 9554 2PEPYC 9654 2P19.8311.3111.26106.271977.0810680.95157.758543.8116.416921.837971398.322445.492449.29262.8456.6010.47115.2032.5233.79293.4916532.4031482.78155.155537.8216.508891.86415242.64857.32824.93380.606.653.45120.4033.4033.97334.8916941.9930938.50218.692562.2184.708491.78296232.01834.97819.76333.516.443.47175.7439.4738.77369.6418452.0235643.79246.700596.9084.052371.68307159.13706.83719.49302.455.923.0782.0438.2438.53155.288039.5715659.32132.721495.2407.647352.01949389.53832.02825.52205.713.978.09101.5643.6443.40191.419843.0319170.61103.184435.1949.689042.30611471.821092.971098.96250.394.869.95OpenBenchmarking.org

CPU Peak Freq (Highest CPU Core Frequency) Monitor

Phoronix Test Suite System Monitoring

OpenBenchmarking.orgMegahertzCPU Peak Freq (Highest CPU Core Frequency) MonitorPhoronix Test Suite System MonitoringXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyXeon Max 9480 2P, No HBMXeon Max 9480 2P, No HBM, Max AVX512 FP166001200180024003000Min: 1100 / Avg: 3207.85 / Max: 3577Min: 1566 / Avg: 3308.69 / Max: 3551Min: 1000 / Avg: 3056.85 / Max: 3576Min: 1800 / Avg: 3336.55 / Max: 3518

CPU Power Consumption Monitor

Phoronix Test Suite System Monitoring

OpenBenchmarking.orgWattsCPU Power Consumption MonitorPhoronix Test Suite System MonitoringEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyXeon Max 9480 2P, No HBMXeon Max 9480 2P, No HBM, Max AVX512 FP162004006008001000Min: 34.39 / Avg: 413.82 / Max: 710.63Min: 31.88 / Avg: 448.81 / Max: 725.51Min: 87.33 / Avg: 548.37 / Max: 899.87Min: 129.15 / Avg: 563.95 / Max: 1026.74Min: 93.37 / Avg: 522.99 / Max: 664.03Min: 166.66 / Avg: 528.06 / Max: 636.62

System Power Consumption Monitor

Phoronix Test Suite System Monitoring

OpenBenchmarking.orgWattsSystem Power Consumption MonitorPhoronix Test Suite System MonitoringXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyXeon Max 9480 2P, No HBMXeon Max 9480 2P, No HBM, Max AVX512 FP16160320480640800Min: 260 / Avg: 751.74 / Max: 907Min: 250 / Avg: 752.26 / Max: 929Min: 251 / Avg: 735.78 / Max: 904Min: 276 / Avg: 733.34 / Max: 937

OpenVINO

Model: Face Detection FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Face Detection FP16 - Device: CPUEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyXeon Max 9480 2P, No HBMXeon Max 9480 2P, No HBM, Max AVX512 FP164080120160200SE +/- 0.04, N = 3SE +/- 0.25, N = 3SE +/- 0.41, N = 3SE +/- 0.12, N = 3SE +/- 0.29, N = 3SE +/- 0.11, N = 382.04101.56120.40175.74115.2019.831. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

Model: Person Detection FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Person Detection FP16 - Device: CPUEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyXeon Max 9480 2P, No HBMXeon Max 9480 2P, No HBM, Max AVX512 FP161020304050SE +/- 0.23, N = 3SE +/- 0.21, N = 3SE +/- 0.24, N = 3SE +/- 0.08, N = 3SE +/- 0.42, N = 3SE +/- 0.04, N = 338.2443.6433.4039.4732.5211.311. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

Model: Person Detection FP32 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Person Detection FP32 - Device: CPUEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyXeon Max 9480 2P, No HBMXeon Max 9480 2P, No HBM, Max AVX512 FP161020304050SE +/- 0.04, N = 3SE +/- 0.20, N = 3SE +/- 0.31, N = 3SE +/- 0.42, N = 3SE +/- 0.25, N = 3SE +/- 0.07, N = 338.5343.4033.9738.7733.7911.261. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

Model: Face Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyXeon Max 9480 2P, No HBMXeon Max 9480 2P, No HBM, Max AVX512 FP1680160240320400SE +/- 0.14, N = 3SE +/- 0.06, N = 3SE +/- 0.45, N = 3SE +/- 0.06, N = 3SE +/- 0.25, N = 3SE +/- 0.10, N = 3155.28191.41334.89369.64293.49106.271. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

Model: Weld Porosity Detection FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16 - Device: CPUEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyXeon Max 9480 2P, No HBMXeon Max 9480 2P, No HBM, Max AVX512 FP164K8K12K16K20KSE +/- 1.22, N = 3SE +/- 2.32, N = 3SE +/- 13.63, N = 3SE +/- 33.75, N = 3SE +/- 65.25, N = 3SE +/- 2.85, N = 38039.579843.0316941.9918452.0216532.401977.081. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

Model: Weld Porosity Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyXeon Max 9480 2P, No HBMXeon Max 9480 2P, No HBM, Max AVX512 FP168K16K24K32K40KSE +/- 3.60, N = 3SE +/- 4.81, N = 3SE +/- 38.32, N = 3SE +/- 37.65, N = 3SE +/- 37.83, N = 3SE +/- 12.56, N = 315659.3219170.6130938.5035643.7931482.7810680.951. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

ONNX Runtime

Model: GPT-2 - Device: CPU - Executor: Standard

OpenBenchmarking.orgInferences Per Second, More Is BetterONNX Runtime 1.14Model: GPT-2 - Device: CPU - Executor: StandardEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyXeon Max 9480 2P, No HBMXeon Max 9480 2P, No HBM, Max AVX512 FP1650100150200250SE +/- 4.25, N = 15SE +/- 0.03, N = 3SE +/- 9.23, N = 15SE +/- 3.11, N = 3SE +/- 4.09, N = 15SE +/- 4.54, N = 15132.72103.18218.69246.70155.16157.761. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

ONNX Runtime

Model: CaffeNet 12-int8 - Device: CPU - Executor: Standard

OpenBenchmarking.orgInferences Per Second, More Is BetterONNX Runtime 1.14Model: CaffeNet 12-int8 - Device: CPU - Executor: StandardEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyXeon Max 9480 2P, No HBMXeon Max 9480 2P, No HBM, Max AVX512 FP16130260390520650SE +/- 5.94, N = 4SE +/- 7.41, N = 15SE +/- 8.12, N = 15SE +/- 10.95, N = 15SE +/- 7.97, N = 15SE +/- 3.29, N = 3495.24435.19562.22596.91537.82543.811. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxXeon Max 9480 2P, HBM Caching120017403505Xeon Max 9480 2P, HBM Only200034933508Xeon Max 9480 2P, No HBM110016543503Xeon Max 9480 2P, No HBM, Max AVX512 FP16350035003512OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxXeon Max 9480 2P, HBM Caching190030233512Xeon Max 9480 2P, HBM Only350035003517Xeon Max 9480 2P, No HBM180033133518Xeon Max 9480 2P, No HBM, Max AVX512 FP16220034813515OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxXeon Max 9480 2P, HBM Caching190030743506Xeon Max 9480 2P, HBM Only264634963508Xeon Max 9480 2P, No HBM208733963518Xeon Max 9480 2P, No HBM, Max AVX512 FP16350035003511OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxXeon Max 9480 2P, HBM Caching170023703505Xeon Max 9480 2P, HBM Only190033593505Xeon Max 9480 2P, No HBM150021953515Xeon Max 9480 2P, No HBM, Max AVX512 FP16180024923510OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxXeon Max 9480 2P, HBM Caching170019003500Xeon Max 9480 2P, HBM Only190026183506Xeon Max 9480 2P, No HBM160018463500Xeon Max 9480 2P, No HBM, Max AVX512 FP16350035003509OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxXeon Max 9480 2P, HBM Caching150019023504Xeon Max 9480 2P, HBM Only180029323508Xeon Max 9480 2P, No HBM160019343502Xeon Max 9480 2P, No HBM, Max AVX512 FP16180020233500OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

ONNX Runtime

CPU Peak Freq (Highest CPU Core Frequency) Monitor

OpenBenchmarking.orgMegahertz, More Is BetterONNX Runtime 1.14CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, No HBM, Max AVX512 FP166001200180024003000Min: 3500 / Avg: 3501.18 / Max: 3518

ONNX Runtime

CPU Peak Freq (Highest CPU Core Frequency) Monitor

OpenBenchmarking.orgMegahertz, More Is BetterONNX Runtime 1.14CPU Peak Freq (Highest CPU Core Frequency) MonitorXeon Max 9480 2P, No HBM, Max AVX512 FP1680016002400320040003500

ONNX Runtime

Model: GPT-2 - Device: CPU - Executor: Standard

OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: GPT-2 - Device: CPU - Executor: StandardEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyXeon Max 9480 2P, No HBMXeon Max 9480 2P, No HBM, Max AVX512 FP163691215SE +/- 0.25707, N = 15SE +/- 0.00311, N = 3SE +/- 0.23765, N = 15SE +/- 0.05143, N = 3SE +/- 0.18098, N = 15SE +/- 0.19970, N = 157.647359.689044.708494.052376.508896.416921. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

ONNX Runtime

Model: CaffeNet 12-int8 - Device: CPU - Executor: Standard

OpenBenchmarking.orgInference Time Cost (ms), Fewer Is BetterONNX Runtime 1.14Model: CaffeNet 12-int8 - Device: CPU - Executor: StandardEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyXeon Max 9480 2P, No HBMXeon Max 9480 2P, No HBM, Max AVX512 FP160.51891.03781.55672.07562.5945SE +/- 0.02472, N = 4SE +/- 0.03734, N = 15SE +/- 0.02850, N = 15SE +/- 0.03485, N = 15SE +/- 0.02958, N = 15SE +/- 0.01109, N = 32.019492.306111.782961.683071.864151.837971. (CXX) g++ options: -ffunction-sections -fdata-sections -march=native -mtune=native -O3 -flto=auto -fno-fat-lto-objects -ldl -lrt

OpenVINO

Model: Face Detection FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Face Detection FP16 - Device: CPUEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyXeon Max 9480 2P, No HBMXeon Max 9480 2P, No HBM, Max AVX512 FP1630060090012001500SE +/- 0.19, N = 3SE +/- 0.93, N = 3SE +/- 0.80, N = 3SE +/- 0.10, N = 3SE +/- 0.61, N = 3SE +/- 8.00, N = 3389.53471.82232.01159.13242.641398.32MIN: 380.82 / MAX: 432.57MIN: 431.51 / MAX: 569.3MIN: 943.82 / MAX: 1957.071. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

Model: Person Detection FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Person Detection FP16 - Device: CPUEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyXeon Max 9480 2P, No HBMXeon Max 9480 2P, No HBM, Max AVX512 FP165001000150020002500SE +/- 5.30, N = 3SE +/- 5.11, N = 3SE +/- 6.17, N = 3SE +/- 1.42, N = 3SE +/- 10.44, N = 3SE +/- 9.42, N = 3832.021092.97834.97706.83857.322445.49MIN: 743.95 / MAX: 1258.17MIN: 825.28 / MAX: 1814.82MIN: 558.93 / MAX: 2258.39MIN: 455.92 / MAX: 1885.53MIN: 481.73 / MAX: 1422.59MIN: 1611.66 / MAX: 3114.741. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

Model: Person Detection FP32 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Person Detection FP32 - Device: CPUEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyXeon Max 9480 2P, No HBMXeon Max 9480 2P, No HBM, Max AVX512 FP165001000150020002500SE +/- 1.07, N = 3SE +/- 5.33, N = 3SE +/- 7.20, N = 3SE +/- 7.70, N = 3SE +/- 6.05, N = 3SE +/- 11.98, N = 3825.521098.96819.76719.49824.932449.29MIN: 747.05 / MAX: 1238.39MIN: 825.62 / MAX: 1835.76MIN: 516.88 / MAX: 2598.39MIN: 473.54 / MAX: 2264.09MIN: 445.3 / MAX: 1462.45MIN: 1558.62 / MAX: 3306.731. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

Model: Face Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyXeon Max 9480 2P, No HBMXeon Max 9480 2P, No HBM, Max AVX512 FP1680160240320400SE +/- 0.22, N = 3SE +/- 0.04, N = 3SE +/- 0.44, N = 3SE +/- 0.04, N = 3SE +/- 0.34, N = 3SE +/- 0.33, N = 3205.71250.39333.51302.45380.60262.84MIN: 202.38 / MAX: 252.14MIN: 222.36 / MAX: 303.72MIN: 285.32 / MAX: 516.28MIN: 217.26 / MAX: 450.48MIN: 286.27 / MAX: 602.06MIN: 192.97 / MAX: 452.591. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

Model: Weld Porosity Detection FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16 - Device: CPUEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyXeon Max 9480 2P, No HBMXeon Max 9480 2P, No HBM, Max AVX512 FP161326395265SE +/- 0.00, N = 3SE +/- 0.00, N = 3SE +/- 0.01, N = 3SE +/- 0.01, N = 3SE +/- 0.03, N = 3SE +/- 0.08, N = 33.974.866.445.926.6556.60MIN: 44.94 / MAX: 112.121. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

Model: Weld Porosity Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUEPYC 9554 2PEPYC 9654 2PXeon Max 9480 2P, HBM CachingXeon Max 9480 2P, HBM OnlyXeon Max 9480 2P, No HBMXeon Max 9480 2P, No HBM, Max AVX512 FP163691215SE +/- 0.00, N = 3SE +/- 0.00, N = 3SE +/- 0.00, N = 3SE +/- 0.01, N = 3SE +/- 0.00, N = 3SE +/- 0.01, N = 38.099.953.473.073.4510.47MIN: 7.89 / MAX: 45.39MIN: 8.36 / MAX: 40.39MIN: 2.48 / MAX: 56.57MIN: 2.48 / MAX: 68.92MIN: 2.47 / MAX: 46.27MIN: 7.24 / MAX: 56.731. (CXX) g++ options: -isystem -fsigned-char -ffunction-sections -fdata-sections -msse4.1 -msse4.2 -O3 -fno-strict-overflow -fwrapv -fPIC -fvisibility=hidden -Os -std=c++11 -MD -MT -MF

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxEPYC 9554 2P49638711EPYC 9654 2P58646726Xeon Max 9480 2P, HBM Caching136533628Xeon Max 9480 2P, HBM Only256574649Xeon Max 9480 2P, No HBM163519586Xeon Max 9480 2P, No HBM, Max AVX512 FP16259561637OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxEPYC 9554 2P49596683EPYC 9654 2P60608699Xeon Max 9480 2P, HBM Caching204538618Xeon Max 9480 2P, HBM Only252563639Xeon Max 9480 2P, No HBM205515616Xeon Max 9480 2P, No HBM, Max AVX512 FP16267546628OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxEPYC 9554 2P49600684EPYC 9654 2P60606703Xeon Max 9480 2P, HBM Caching228547624Xeon Max 9480 2P, HBM Only223567643Xeon Max 9480 2P, No HBM245531618Xeon Max 9480 2P, No HBM, Max AVX512 FP16241553629OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxEPYC 9554 2P50606675EPYC 9654 2P59613687Xeon Max 9480 2P, HBM Caching168531616Xeon Max 9480 2P, HBM Only249561682Xeon Max 9480 2P, No HBM157500632Xeon Max 9480 2P, No HBM, Max AVX512 FP16184553616OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxEPYC 9554 2P48648698EPYC 9654 2P57655705Xeon Max 9480 2P, HBM Caching163557629Xeon Max 9480 2P, HBM Only235582661Xeon Max 9480 2P, No HBM165538595Xeon Max 9480 2P, No HBM, Max AVX512 FP16268564605OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxEPYC 9554 2P49623675EPYC 9654 2P59632685Xeon Max 9480 2P, HBM Caching137552614Xeon Max 9480 2P, HBM Only256583675Xeon Max 9480 2P, No HBM161541594Xeon Max 9480 2P, No HBM, Max AVX512 FP16168556603OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

System Power Consumption Monitor

MinAvgMaxXeon Max 9480 2P, HBM Caching281724839Xeon Max 9480 2P, HBM Only362757833Xeon Max 9480 2P, No HBM274734840Xeon Max 9480 2P, No HBM, Max AVX512 FP16399769937OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor2004006008001000

OpenVINO

System Power Consumption Monitor

MinAvgMaxXeon Max 9480 2P, HBM Caching298736817Xeon Max 9480 2P, HBM Only360745835Xeon Max 9480 2P, No HBM295737874Xeon Max 9480 2P, No HBM, Max AVX512 FP16408757822OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor2004006008001000

OpenVINO

System Power Consumption Monitor

MinAvgMaxXeon Max 9480 2P, HBM Caching369748857Xeon Max 9480 2P, HBM Only373752832Xeon Max 9480 2P, No HBM378758872Xeon Max 9480 2P, No HBM, Max AVX512 FP16413768860OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor2004006008001000

OpenVINO

System Power Consumption Monitor

MinAvgMaxXeon Max 9480 2P, HBM Caching270722818Xeon Max 9480 2P, HBM Only365742888Xeon Max 9480 2P, No HBM256717841Xeon Max 9480 2P, No HBM, Max AVX512 FP16307762847OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor2004006008001000

OpenVINO

System Power Consumption Monitor

MinAvgMaxXeon Max 9480 2P, HBM Caching279756840Xeon Max 9480 2P, HBM Only342768875Xeon Max 9480 2P, No HBM268759840Xeon Max 9480 2P, No HBM, Max AVX512 FP16420774811OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor2004006008001000

OpenVINO

System Power Consumption Monitor

MinAvgMaxXeon Max 9480 2P, HBM Caching275752830Xeon Max 9480 2P, HBM Only370768854Xeon Max 9480 2P, No HBM284753808Xeon Max 9480 2P, No HBM, Max AVX512 FP16276765811OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3System Power Consumption Monitor2004006008001000

ONNX Runtime

System Power Consumption Monitor

OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14System Power Consumption MonitorXeon Max 9480 2P, No HBM, Max AVX512 FP16130260390520650Min: 386 / Avg: 718.32 / Max: 749

ONNX Runtime

System Power Consumption Monitor

MinAvgMaxXeon Max 9480 2P, No HBM, Max AVX512 FP16416765857OpenBenchmarking.orgWatts, Fewer Is BetterONNX Runtime 1.14System Power Consumption Monitor2004006008001000

Phoronix Test Suite v10.8.5