OpenVINO AMX Benchmarks

Benchmarks by Michael Larabel for a future article.

HTML result view exported from: https://openbenchmarking.org/result/2301160-NE-OPENVINOA07&sro.

OpenVINO AMX BenchmarksProcessorMotherboardChipsetMemoryDiskGraphicsMonitorNetworkOSKernelDesktopDisplay ServerVulkanCompilerFile-SystemScreen ResolutionAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads)Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS)Intel Device 1bce16 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96ASPEEDVGA HDMI4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-TUbuntu 22.046.1.4-060104-generic (x86_64)GNOME Shell 42.2X Server 1.21.1.31.2.204GCC 11.3.0 + Clang 14.0.0-1ubuntu1ext41920x1080OpenBenchmarking.orgKernel Details- Transparent Huge Pages: madviseCompiler Details- --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details- Scaling Governor: intel_pstate performance (EPP: performance) - CPU Microcode: 0x2b0000c0Python Details- Python 3.10.6Security Details- itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

OpenVINO AMX Benchmarksopenvino: Face Detection FP16-INT8 - CPUopenvino: Face Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUopenvino: Person Vehicle Bike Detection FP16 - CPUopenvino: Person Vehicle Bike Detection FP16 - CPUopenvino: Machine Translation EN To DE FP16 - CPUopenvino: Machine Translation EN To DE FP16 - CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE282.25424.1623066.635.184778.3425.09724.1082.79146.08409.818897.1013.462730.0721.95321.28186.53146.06409.848918.6213.432767.1621.66314.61190.49146.02409.958909.0513.441821.0932.93387.62154.5563.21942.795835.3920.541820.2732.94378.72158.26OpenBenchmarking.org

OpenVINO

Model: Face Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUAVX512_COREAVX512_CORE_AMXAVX512_CORE_BF16AVX512_CORE_FP16AVX512_CORE_VNNI60120180240300SE +/- 0.03, N = 3SE +/- 2.56, N = 3SE +/- 0.22, N = 3SE +/- 0.29, N = 3SE +/- 0.30, N = 363.21282.25146.06146.08146.021. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

Model: Face Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUAVX512_COREAVX512_CORE_AMXAVX512_CORE_BF16AVX512_CORE_FP16AVX512_CORE_VNNI2004006008001000SE +/- 0.30, N = 3SE +/- 3.75, N = 3SE +/- 0.65, N = 3SE +/- 0.81, N = 3SE +/- 0.81, N = 3942.79424.16409.84409.81409.95MIN: 383.12 / MAX: 1157.46MIN: 153.43 / MAX: 1952.95MIN: 208.02 / MAX: 643.02MIN: 203.37 / MAX: 909.92MIN: 216.25 / MAX: 615.531. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxAVX512_CORE190026493515AVX512_CORE_AMX190025613507AVX512_CORE_BF16190024013505AVX512_CORE_FP16123123743508AVX512_CORE_VNNI112623933512OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxAVX512_CORE214643759AVX512_CORE_AMX208616761AVX512_CORE_BF16104641758AVX512_CORE_FP16210641748AVX512_CORE_VNNI201640752OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

CPU Temperature Monitor

MinAvgMaxAVX512_CORE33.052.157.0AVX512_CORE_AMX30.048.055.0AVX512_CORE_BF1631.051.156.0AVX512_CORE_FP1634.052.357.0AVX512_CORE_VNNI35.052.659.0OpenBenchmarking.orgCelsius, Fewer Is BetterOpenVINO 2022.3CPU Temperature Monitor1632486480

OpenVINO

Model: Weld Porosity Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUAVX512_COREAVX512_CORE_AMXAVX512_CORE_BF16AVX512_CORE_FP16AVX512_CORE_VNNI5K10K15K20K25KSE +/- 78.19, N = 3SE +/- 92.31, N = 3SE +/- 85.46, N = 3SE +/- 24.48, N = 3SE +/- 52.22, N = 35835.3923066.638918.628897.108909.051. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

Model: Weld Porosity Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUAVX512_COREAVX512_CORE_AMXAVX512_CORE_BF16AVX512_CORE_FP16AVX512_CORE_VNNI510152025SE +/- 0.28, N = 3SE +/- 0.02, N = 3SE +/- 0.13, N = 3SE +/- 0.04, N = 3SE +/- 0.08, N = 320.545.1813.4313.4613.44MIN: 9.17 / MAX: 114MIN: 1.43 / MAX: 102.62MIN: 3.72 / MAX: 102.63MIN: 3.82 / MAX: 88.11MIN: 3.47 / MAX: 112.161. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxAVX512_CORE190026743503AVX512_CORE_AMX190028603501AVX512_CORE_BF16190028913512AVX512_CORE_FP16190029283512AVX512_CORE_VNNI190029033511OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxAVX512_CORE212668759AVX512_CORE_AMX217663757AVX512_CORE_BF16210667769AVX512_CORE_FP16210665756AVX512_CORE_VNNI211665760OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

CPU Temperature Monitor

MinAvgMaxAVX512_CORE39.054.958.0AVX512_CORE_AMX39.052.255.0AVX512_CORE_BF1638.054.658.0AVX512_CORE_FP1640.055.258.0AVX512_CORE_VNNI38.054.858.0OpenBenchmarking.orgCelsius, Fewer Is BetterOpenVINO 2022.3CPU Temperature Monitor1632486480

OpenVINO

Model: Person Vehicle Bike Detection FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Person Vehicle Bike Detection FP16 - Device: CPUAVX512_COREAVX512_CORE_AMXAVX512_CORE_BF16AVX512_CORE_FP16AVX512_CORE_VNNI10002000300040005000SE +/- 8.67, N = 3SE +/- 14.65, N = 3SE +/- 16.62, N = 3SE +/- 11.17, N = 3SE +/- 7.34, N = 31820.274778.342767.162730.071821.091. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

Model: Person Vehicle Bike Detection FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Person Vehicle Bike Detection FP16 - Device: CPUAVX512_COREAVX512_CORE_AMXAVX512_CORE_BF16AVX512_CORE_FP16AVX512_CORE_VNNI816243240SE +/- 0.16, N = 3SE +/- 0.08, N = 3SE +/- 0.13, N = 3SE +/- 0.09, N = 3SE +/- 0.13, N = 332.9425.0921.6621.9532.93MIN: 12.45 / MAX: 111.25MIN: 11.66 / MAX: 132.09MIN: 13.42 / MAX: 101.56MIN: 11.62 / MAX: 120.04MIN: 12.7 / MAX: 103.821. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxAVX512_CORE190029103513AVX512_CORE_AMX190029653501AVX512_CORE_BF16190027553513AVX512_CORE_FP16190027393511AVX512_CORE_VNNI190029343508OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxAVX512_CORE212655749AVX512_CORE_AMX213640755AVX512_CORE_BF16210652754AVX512_CORE_FP16132653756AVX512_CORE_VNNI210655748OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

CPU Temperature Monitor

MinAvgMaxAVX512_CORE40.052.155.0AVX512_CORE_AMX39.051.955.0AVX512_CORE_BF1639.052.557.0AVX512_CORE_FP1639.052.857.0AVX512_CORE_VNNI39.052.055.0OpenBenchmarking.orgCelsius, Fewer Is BetterOpenVINO 2022.3CPU Temperature Monitor1632486480

OpenVINO

Model: Machine Translation EN To DE FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Machine Translation EN To DE FP16 - Device: CPUAVX512_COREAVX512_CORE_AMXAVX512_CORE_BF16AVX512_CORE_FP16AVX512_CORE_VNNI160320480640800SE +/- 5.28, N = 3SE +/- 5.21, N = 3SE +/- 3.03, N = 3SE +/- 0.77, N = 3SE +/- 2.49, N = 3378.72724.10314.61321.28387.621. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

Model: Machine Translation EN To DE FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Machine Translation EN To DE FP16 - Device: CPUAVX512_COREAVX512_CORE_AMXAVX512_CORE_BF16AVX512_CORE_FP16AVX512_CORE_VNNI4080120160200SE +/- 2.22, N = 3SE +/- 0.60, N = 3SE +/- 1.85, N = 3SE +/- 0.45, N = 3SE +/- 1.04, N = 3158.2682.79190.49186.53154.55MIN: 62.25 / MAX: 1819.89MIN: 31.64 / MAX: 1588.97MIN: 117.21 / MAX: 1606.21MIN: 92.82 / MAX: 808.7MIN: 90.51 / MAX: 1925.111. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxAVX512_CORE190025903518AVX512_CORE_AMX190027923516AVX512_CORE_BF16190027103512AVX512_CORE_FP16190026563508AVX512_CORE_VNNI190025683509OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxAVX512_CORE211648753AVX512_CORE_AMX212644742AVX512_CORE_BF16209650756AVX512_CORE_FP16211652756AVX512_CORE_VNNI208648751OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

CPU Temperature Monitor

MinAvgMaxAVX512_CORE40.052.456.0AVX512_CORE_AMX39.051.555.0AVX512_CORE_BF1639.052.657.0AVX512_CORE_FP1638.052.656.0AVX512_CORE_VNNI39.052.456.0OpenBenchmarking.orgCelsius, Fewer Is BetterOpenVINO 2022.3CPU Temperature Monitor1632486480

Phoronix Test Suite v10.8.4