OpenVINO AMX Benchmarks AMD EPYC Genoa

Benchmarks by Michael Larabel for a future article.

HTML result view exported from: https://openbenchmarking.org/result/2301166-NE-OPENVINOA64&gru&export=pdf&sro&rro.

OpenVINO AMX Benchmarks AMD EPYC GenoaProcessorMotherboardChipsetMemoryDiskGraphicsMonitorNetworkOSKernelDesktopDisplay ServerVulkanCompilerFile-SystemScreen ResolutionXeon Platinum 8490H 2PXeon Platinum 8490H 2P - No AMXEPYC 9654 2P2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads)Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS)Intel Device 1bce16 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96ASPEEDVGA HDMI4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-TUbuntu 22.046.1.4-060104-generic (x86_64)GNOME Shell 42.2X Server 1.21.1.31.2.204GCC 11.3.0 + Clang 14.0.0-1ubuntu1ext41920x10802 x AMD EPYC 9654 96-Core @ 2.40GHz (192 Cores / 384 Threads)AMD Titanite_4G (RTI1002E BIOS)AMD Device 14a424 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007Broadcom NetXtreme BCM5720 PCIeOpenBenchmarking.orgKernel Details- Transparent Huge Pages: madviseCompiler Details- --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details- Xeon Platinum 8490H 2P: Scaling Governor: intel_pstate performance (EPP: performance) - CPU Microcode: 0x2b0000c0- Xeon Platinum 8490H 2P - No AMX: Scaling Governor: intel_pstate performance (EPP: performance) - CPU Microcode: 0x2b0000c0- EPYC 9654 2P: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa10110dPython Details- Python 3.10.6Security Details- Xeon Platinum 8490H 2P: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected - Xeon Platinum 8490H 2P - No AMX: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected - EPYC 9654 2P: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected

OpenVINO AMX Benchmarks AMD EPYC Genoaopenvino: Face Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUopenvino: Person Vehicle Bike Detection FP16 - CPUopenvino: Machine Translation EN To DE FP16 - CPUopenvino: Face Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUopenvino: Person Vehicle Bike Detection FP16 - CPUopenvino: Machine Translation EN To DE FP16 - CPUXeon Platinum 8490H 2PXeon Platinum 8490H 2P - No AMXEPYC 9654 2P282.2523066.634778.34724.10424.165.1825.0982.79146.088897.102730.07321.28409.8113.4621.95186.53188.6418678.165320.83989.16507.2510.2618.0397.00OpenBenchmarking.org

CPU Peak Freq (Highest CPU Core Frequency) Monitor

Phoronix Test Suite System Monitoring

OpenBenchmarking.orgMegahertzCPU Peak Freq (Highest CPU Core Frequency) MonitorPhoronix Test Suite System MonitoringXeon Platinum 8490H 2P - No AMXXeon Platinum 8490H 2PEPYC 9654 2P7001400210028003500Min: 1000 / Avg: 2603.67 / Max: 3512Min: 1900 / Avg: 2744.33 / Max: 3516Min: 2400 / Avg: 3129.46 / Max: 3763

CPU Power Consumption Monitor

Phoronix Test Suite System Monitoring

OpenBenchmarking.orgWattsCPU Power Consumption MonitorPhoronix Test Suite System MonitoringXeon Platinum 8490H 2P - No AMXXeon Platinum 8490H 2PEPYC 9654 2P130260390520650Min: 122.95 / Avg: 646.09 / Max: 763.9Min: 208.07 / Avg: 624.35 / Max: 760.79Min: 53.42 / Avg: 596.25 / Max: 739.81

OpenVINO

Model: Face Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUXeon Platinum 8490H 2P - No AMXXeon Platinum 8490H 2PEPYC 9654 2P60120180240300SE +/- 0.29, N = 3SE +/- 2.56, N = 3SE +/- 0.17, N = 3146.08282.25188.641. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

Model: Weld Porosity Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUXeon Platinum 8490H 2P - No AMXXeon Platinum 8490H 2PEPYC 9654 2P5K10K15K20K25KSE +/- 24.48, N = 3SE +/- 92.31, N = 3SE +/- 10.84, N = 38897.1023066.6318678.161. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

Model: Person Vehicle Bike Detection FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Person Vehicle Bike Detection FP16 - Device: CPUXeon Platinum 8490H 2P - No AMXXeon Platinum 8490H 2PEPYC 9654 2P11002200330044005500SE +/- 11.17, N = 3SE +/- 14.65, N = 3SE +/- 44.61, N = 82730.074778.345320.831. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

Model: Machine Translation EN To DE FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Machine Translation EN To DE FP16 - Device: CPUXeon Platinum 8490H 2P - No AMXXeon Platinum 8490H 2PEPYC 9654 2P2004006008001000SE +/- 0.77, N = 3SE +/- 5.21, N = 3SE +/- 8.13, N = 8321.28724.10989.161. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxXeon Platinum 8490H 2P - No AMX123123743508Xeon Platinum 8490H 2P190025613507EPYC 9654 2P240029903702OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxXeon Platinum 8490H 2P - No AMX190029283512Xeon Platinum 8490H 2P190028603501EPYC 9654 2P240028093702OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxXeon Platinum 8490H 2P - No AMX190027393511Xeon Platinum 8490H 2P190029653501EPYC 9654 2P240034933763OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxXeon Platinum 8490H 2P - No AMX190026563508Xeon Platinum 8490H 2P190027923516EPYC 9654 2P240029653700OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

Model: Face Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUXeon Platinum 8490H 2P - No AMXXeon Platinum 8490H 2PEPYC 9654 2P110220330440550SE +/- 0.81, N = 3SE +/- 3.75, N = 3SE +/- 0.31, N = 3409.81424.16507.25MIN: 203.37 / MAX: 909.92MIN: 153.43 / MAX: 1952.95MIN: 231.44 / MAX: 623.691. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

Model: Weld Porosity Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUXeon Platinum 8490H 2P - No AMXXeon Platinum 8490H 2PEPYC 9654 2P3691215SE +/- 0.04, N = 3SE +/- 0.02, N = 3SE +/- 0.01, N = 313.465.1810.26MIN: 3.82 / MAX: 88.11MIN: 1.43 / MAX: 102.62MIN: 4.21 / MAX: 95.331. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

Model: Person Vehicle Bike Detection FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Person Vehicle Bike Detection FP16 - Device: CPUXeon Platinum 8490H 2P - No AMXXeon Platinum 8490H 2PEPYC 9654 2P612182430SE +/- 0.09, N = 3SE +/- 0.08, N = 3SE +/- 0.15, N = 821.9525.0918.03MIN: 11.62 / MAX: 120.04MIN: 11.66 / MAX: 132.09MIN: 5.34 / MAX: 112.511. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

Model: Machine Translation EN To DE FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Machine Translation EN To DE FP16 - Device: CPUXeon Platinum 8490H 2P - No AMXXeon Platinum 8490H 2PEPYC 9654 2P4080120160200SE +/- 0.45, N = 3SE +/- 0.60, N = 3SE +/- 0.78, N = 8186.5382.7997.00MIN: 92.82 / MAX: 808.7MIN: 31.64 / MAX: 1588.97MIN: 41.26 / MAX: 1899.381. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxXeon Platinum 8490H 2P - No AMX210641748Xeon Platinum 8490H 2P208616761EPYC 9654 2P54602717OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxXeon Platinum 8490H 2P - No AMX210665756Xeon Platinum 8490H 2P217663757EPYC 9654 2P59652717OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxXeon Platinum 8490H 2P - No AMX132653756Xeon Platinum 8490H 2P213640755EPYC 9654 2P55567643OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxXeon Platinum 8490H 2P - No AMX211652756Xeon Platinum 8490H 2P212644742EPYC 9654 2P55650740OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

Phoronix Test Suite v10.8.5