OpenVINO AMX Benchmarks AMD EPYC Genoa

Benchmarks by Michael Larabel for a future article.

HTML result view exported from: https://openbenchmarking.org/result/2301166-NE-OPENVINOA64&export=txt&grs&rdt&rro.

OpenVINO AMX Benchmarks AMD EPYC GenoaProcessorMotherboardChipsetMemoryDiskGraphicsMonitorNetworkOSKernelDesktopDisplay ServerVulkanCompilerFile-SystemScreen ResolutionXeon Platinum 8490H 2PXeon Platinum 8490H 2P - No AMXEPYC 9654 2P2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads)Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS)Intel Device 1bce16 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96ASPEEDVGA HDMI4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-TUbuntu 22.046.1.4-060104-generic (x86_64)GNOME Shell 42.2X Server 1.21.1.31.2.204GCC 11.3.0 + Clang 14.0.0-1ubuntu1ext41920x10802 x AMD EPYC 9654 96-Core @ 2.40GHz (192 Cores / 384 Threads)AMD Titanite_4G (RTI1002E BIOS)AMD Device 14a424 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007Broadcom NetXtreme BCM5720 PCIeOpenBenchmarking.orgKernel Details- Transparent Huge Pages: madviseCompiler Details- --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details- Xeon Platinum 8490H 2P: Scaling Governor: intel_pstate performance (EPP: performance) - CPU Microcode: 0x2b0000c0- Xeon Platinum 8490H 2P - No AMX: Scaling Governor: intel_pstate performance (EPP: performance) - CPU Microcode: 0x2b0000c0- EPYC 9654 2P: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa10110dPython Details- Python 3.10.6Security Details- Xeon Platinum 8490H 2P: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected - Xeon Platinum 8490H 2P - No AMX: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected - EPYC 9654 2P: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected

OpenVINO AMX Benchmarks AMD EPYC Genoaopenvino: Machine Translation EN To DE FP16 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUopenvino: Machine Translation EN To DE FP16 - CPUopenvino: Person Vehicle Bike Detection FP16 - CPUopenvino: Face Detection FP16-INT8 - CPUopenvino: Person Vehicle Bike Detection FP16 - CPUopenvino: Face Detection FP16-INT8 - CPUXeon Platinum 8490H 2PXeon Platinum 8490H 2P - No AMXEPYC 9654 2P724.105.1823066.6382.794778.34282.2525.09424.16321.2813.468897.10186.532730.07146.0821.95409.81989.1610.2618678.1697.005320.83188.6418.03507.25OpenBenchmarking.org

OpenVINO

Model: Machine Translation EN To DE FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Machine Translation EN To DE FP16 - Device: CPUEPYC 9654 2PXeon Platinum 8490H 2P - No AMXXeon Platinum 8490H 2P2004006008001000SE +/- 8.13, N = 8SE +/- 0.77, N = 3SE +/- 5.21, N = 3989.16321.28724.101. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

Model: Weld Porosity Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUEPYC 9654 2PXeon Platinum 8490H 2P - No AMXXeon Platinum 8490H 2P3691215SE +/- 0.01, N = 3SE +/- 0.04, N = 3SE +/- 0.02, N = 310.2613.465.18MIN: 4.21 / MAX: 95.33MIN: 3.82 / MAX: 88.11MIN: 1.43 / MAX: 102.621. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

Model: Weld Porosity Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUEPYC 9654 2PXeon Platinum 8490H 2P - No AMXXeon Platinum 8490H 2P5K10K15K20K25KSE +/- 10.84, N = 3SE +/- 24.48, N = 3SE +/- 92.31, N = 318678.168897.1023066.631. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

Model: Machine Translation EN To DE FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Machine Translation EN To DE FP16 - Device: CPUEPYC 9654 2PXeon Platinum 8490H 2P - No AMXXeon Platinum 8490H 2P4080120160200SE +/- 0.78, N = 8SE +/- 0.45, N = 3SE +/- 0.60, N = 397.00186.5382.79MIN: 41.26 / MAX: 1899.38MIN: 92.82 / MAX: 808.7MIN: 31.64 / MAX: 1588.971. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

Model: Person Vehicle Bike Detection FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Person Vehicle Bike Detection FP16 - Device: CPUEPYC 9654 2PXeon Platinum 8490H 2P - No AMXXeon Platinum 8490H 2P11002200330044005500SE +/- 44.61, N = 8SE +/- 11.17, N = 3SE +/- 14.65, N = 35320.832730.074778.341. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

Model: Face Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUEPYC 9654 2PXeon Platinum 8490H 2P - No AMXXeon Platinum 8490H 2P60120180240300SE +/- 0.17, N = 3SE +/- 0.29, N = 3SE +/- 2.56, N = 3188.64146.08282.251. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

Model: Person Vehicle Bike Detection FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Person Vehicle Bike Detection FP16 - Device: CPUEPYC 9654 2PXeon Platinum 8490H 2P - No AMXXeon Platinum 8490H 2P612182430SE +/- 0.15, N = 8SE +/- 0.09, N = 3SE +/- 0.08, N = 318.0321.9525.09MIN: 5.34 / MAX: 112.51MIN: 11.62 / MAX: 120.04MIN: 11.66 / MAX: 132.091. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

Model: Face Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUEPYC 9654 2PXeon Platinum 8490H 2P - No AMXXeon Platinum 8490H 2P110220330440550SE +/- 0.31, N = 3SE +/- 0.81, N = 3SE +/- 3.75, N = 3507.25409.81424.16MIN: 231.44 / MAX: 623.69MIN: 203.37 / MAX: 909.92MIN: 153.43 / MAX: 1952.951. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

CPU Power Consumption Monitor

Phoronix Test Suite System Monitoring

OpenBenchmarking.orgWattsCPU Power Consumption MonitorPhoronix Test Suite System MonitoringEPYC 9654 2PXeon Platinum 8490H 2P - No AMXXeon Platinum 8490H 2P130260390520650Min: 53.42 / Avg: 596.25 / Max: 739.81Min: 122.95 / Avg: 646.09 / Max: 763.9Min: 208.07 / Avg: 624.35 / Max: 760.79

CPU Peak Freq (Highest CPU Core Frequency) Monitor

Phoronix Test Suite System Monitoring

OpenBenchmarking.orgMegahertzCPU Peak Freq (Highest CPU Core Frequency) MonitorPhoronix Test Suite System MonitoringEPYC 9654 2PXeon Platinum 8490H 2P - No AMXXeon Platinum 8490H 2P7001400210028003500Min: 2400 / Avg: 3129.46 / Max: 3763Min: 1000 / Avg: 2603.67 / Max: 3512Min: 1900 / Avg: 2744.33 / Max: 3516

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxEPYC 9654 2P55650740Xeon Platinum 8490H 2P - No AMX211652756Xeon Platinum 8490H 2P212644742OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxEPYC 9654 2P240029653700Xeon Platinum 8490H 2P - No AMX190026563508Xeon Platinum 8490H 2P190027923516OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxEPYC 9654 2P55567643Xeon Platinum 8490H 2P - No AMX132653756Xeon Platinum 8490H 2P213640755OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxEPYC 9654 2P240034933763Xeon Platinum 8490H 2P - No AMX190027393511Xeon Platinum 8490H 2P190029653501OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxEPYC 9654 2P59652717Xeon Platinum 8490H 2P - No AMX210665756Xeon Platinum 8490H 2P217663757OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxEPYC 9654 2P240028093702Xeon Platinum 8490H 2P - No AMX190029283512Xeon Platinum 8490H 2P190028603501OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxEPYC 9654 2P54602717Xeon Platinum 8490H 2P - No AMX210641748Xeon Platinum 8490H 2P208616761OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxEPYC 9654 2P240029903702Xeon Platinum 8490H 2P - No AMX123123743508Xeon Platinum 8490H 2P190025613507OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

Phoronix Test Suite v10.8.5