OpenVINO AMX Benchmarks

Benchmarks by Michael Larabel for a future article.

HTML result view exported from: https://openbenchmarking.org/result/2301160-NE-OPENVINOA07.

OpenVINO AMX BenchmarksProcessorMotherboardChipsetMemoryDiskGraphicsMonitorNetworkOSKernelDesktopDisplay ServerVulkanCompilerFile-SystemScreen ResolutionAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads)Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS)Intel Device 1bce16 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96ASPEEDVGA HDMI4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-TUbuntu 22.046.1.4-060104-generic (x86_64)GNOME Shell 42.2X Server 1.21.1.31.2.204GCC 11.3.0 + Clang 14.0.0-1ubuntu1ext41920x1080OpenBenchmarking.orgKernel Details- Transparent Huge Pages: madviseCompiler Details- --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details- Scaling Governor: intel_pstate performance (EPP: performance) - CPU Microcode: 0x2b0000c0Python Details- Python 3.10.6Security Details- itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

OpenVINO AMX Benchmarksopenvino: Face Detection FP16-INT8 - CPUopenvino: Face Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUopenvino: Person Vehicle Bike Detection FP16 - CPUopenvino: Person Vehicle Bike Detection FP16 - CPUopenvino: Machine Translation EN To DE FP16 - CPUopenvino: Machine Translation EN To DE FP16 - CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE282.25424.1623066.635.184778.3425.09724.1082.79146.08409.818897.1013.462730.0721.95321.28186.53146.06409.848918.6213.432767.1621.66314.61190.49146.02409.958909.0513.441821.0932.93387.62154.5563.21942.795835.3920.541820.2732.94378.72158.26OpenBenchmarking.org

OpenVINO

Model: Face Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE60120180240300SE +/- 2.56, N = 3SE +/- 0.29, N = 3SE +/- 0.22, N = 3SE +/- 0.30, N = 3SE +/- 0.03, N = 3282.25146.08146.06146.0263.211. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

Model: Face Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE2004006008001000SE +/- 3.75, N = 3SE +/- 0.81, N = 3SE +/- 0.65, N = 3SE +/- 0.81, N = 3SE +/- 0.30, N = 3424.16409.81409.84409.95942.79MIN: 153.43 / MAX: 1952.95MIN: 203.37 / MAX: 909.92MIN: 208.02 / MAX: 643.02MIN: 216.25 / MAX: 615.53MIN: 383.12 / MAX: 1157.461. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxAVX512_CORE_AMX190025613507AVX512_CORE_FP16123123743508AVX512_CORE_BF16190024013505AVX512_CORE_VNNI112623933512AVX512_CORE190026493515OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxAVX512_CORE_AMX208616761AVX512_CORE_FP16210641748AVX512_CORE_BF16104641758AVX512_CORE_VNNI201640752AVX512_CORE214643759OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

CPU Temperature Monitor

MinAvgMaxAVX512_CORE_AMX30.048.055.0AVX512_CORE_FP1634.052.357.0AVX512_CORE_BF1631.051.156.0AVX512_CORE_VNNI35.052.659.0AVX512_CORE33.052.157.0OpenBenchmarking.orgCelsius, Fewer Is BetterOpenVINO 2022.3CPU Temperature Monitor1632486480

OpenVINO

Model: Weld Porosity Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE5K10K15K20K25KSE +/- 92.31, N = 3SE +/- 24.48, N = 3SE +/- 85.46, N = 3SE +/- 52.22, N = 3SE +/- 78.19, N = 323066.638897.108918.628909.055835.391. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

Model: Weld Porosity Detection FP16-INT8 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE510152025SE +/- 0.02, N = 3SE +/- 0.04, N = 3SE +/- 0.13, N = 3SE +/- 0.08, N = 3SE +/- 0.28, N = 35.1813.4613.4313.4420.54MIN: 1.43 / MAX: 102.62MIN: 3.82 / MAX: 88.11MIN: 3.72 / MAX: 102.63MIN: 3.47 / MAX: 112.16MIN: 9.17 / MAX: 1141. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxAVX512_CORE_AMX190028603501AVX512_CORE_FP16190029283512AVX512_CORE_BF16190028913512AVX512_CORE_VNNI190029033511AVX512_CORE190026743503OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxAVX512_CORE_AMX217663757AVX512_CORE_FP16210665756AVX512_CORE_BF16210667769AVX512_CORE_VNNI211665760AVX512_CORE212668759OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

CPU Temperature Monitor

MinAvgMaxAVX512_CORE_AMX39.052.255.0AVX512_CORE_FP1640.055.258.0AVX512_CORE_BF1638.054.658.0AVX512_CORE_VNNI38.054.858.0AVX512_CORE39.054.958.0OpenBenchmarking.orgCelsius, Fewer Is BetterOpenVINO 2022.3CPU Temperature Monitor1632486480

OpenVINO

Model: Person Vehicle Bike Detection FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Person Vehicle Bike Detection FP16 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE10002000300040005000SE +/- 14.65, N = 3SE +/- 11.17, N = 3SE +/- 16.62, N = 3SE +/- 7.34, N = 3SE +/- 8.67, N = 34778.342730.072767.161821.091820.271. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

Model: Person Vehicle Bike Detection FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Person Vehicle Bike Detection FP16 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE816243240SE +/- 0.08, N = 3SE +/- 0.09, N = 3SE +/- 0.13, N = 3SE +/- 0.13, N = 3SE +/- 0.16, N = 325.0921.9521.6632.9332.94MIN: 11.66 / MAX: 132.09MIN: 11.62 / MAX: 120.04MIN: 13.42 / MAX: 101.56MIN: 12.7 / MAX: 103.82MIN: 12.45 / MAX: 111.251. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxAVX512_CORE_AMX190029653501AVX512_CORE_FP16190027393511AVX512_CORE_BF16190027553513AVX512_CORE_VNNI190029343508AVX512_CORE190029103513OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxAVX512_CORE_AMX213640755AVX512_CORE_FP16132653756AVX512_CORE_BF16210652754AVX512_CORE_VNNI210655748AVX512_CORE212655749OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

CPU Temperature Monitor

MinAvgMaxAVX512_CORE_AMX39.051.955.0AVX512_CORE_FP1639.052.857.0AVX512_CORE_BF1639.052.557.0AVX512_CORE_VNNI39.052.055.0AVX512_CORE40.052.155.0OpenBenchmarking.orgCelsius, Fewer Is BetterOpenVINO 2022.3CPU Temperature Monitor1632486480

OpenVINO

Model: Machine Translation EN To DE FP16 - Device: CPU

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Machine Translation EN To DE FP16 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE160320480640800SE +/- 5.21, N = 3SE +/- 0.77, N = 3SE +/- 3.03, N = 3SE +/- 2.49, N = 3SE +/- 5.28, N = 3724.10321.28314.61387.62378.721. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

Model: Machine Translation EN To DE FP16 - Device: CPU

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Machine Translation EN To DE FP16 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE4080120160200SE +/- 0.60, N = 3SE +/- 0.45, N = 3SE +/- 1.85, N = 3SE +/- 1.04, N = 3SE +/- 2.22, N = 382.79186.53190.49154.55158.26MIN: 31.64 / MAX: 1588.97MIN: 92.82 / MAX: 808.7MIN: 117.21 / MAX: 1606.21MIN: 90.51 / MAX: 1925.11MIN: 62.25 / MAX: 1819.891. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenVINO

CPU Peak Freq (Highest CPU Core Frequency) Monitor

MinAvgMaxAVX512_CORE_AMX190027923516AVX512_CORE_FP16190026563508AVX512_CORE_BF16190027103512AVX512_CORE_VNNI190025683509AVX512_CORE190025903518OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000

OpenVINO

CPU Power Consumption Monitor

MinAvgMaxAVX512_CORE_AMX212644742AVX512_CORE_FP16211652756AVX512_CORE_BF16209650756AVX512_CORE_VNNI208648751AVX512_CORE211648753OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor2004006008001000

OpenVINO

CPU Temperature Monitor

MinAvgMaxAVX512_CORE_AMX39.051.555.0AVX512_CORE_FP1638.052.656.0AVX512_CORE_BF1639.052.657.0AVX512_CORE_VNNI39.052.456.0AVX512_CORE40.052.456.0OpenBenchmarking.orgCelsius, Fewer Is BetterOpenVINO 2022.3CPU Temperature Monitor1632486480

Phoronix Test Suite v10.8.4