Benchmarks by Michael Larabel for a future article.
Compare your own system(s) to this result file with the
Phoronix Test Suite by running the command:
phoronix-test-suite benchmark 2301160-NE-OPENVINOA07 OpenVINO AMX Benchmarks - Phoronix Test Suite OpenVINO AMX Benchmarks Benchmarks by Michael Larabel for a future article.
HTML result view exported from: https://openbenchmarking.org/result/2301160-NE-OPENVINOA07&sro&grt .
OpenVINO AMX Benchmarks Processor Motherboard Chipset Memory Disk Graphics Monitor Network OS Kernel Desktop Display Server Vulkan Compiler File-System Screen Resolution AVX512_CORE_AMX AVX512_CORE_FP16 AVX512_CORE_BF16 AVX512_CORE_VNNI AVX512_CORE 2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads) Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS) Intel Device 1bce 16 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96 ASPEED VGA HDMI 4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-T Ubuntu 22.04 6.1.4-060104-generic (x86_64) GNOME Shell 42.2 X Server 1.21.1.3 1.2.204 GCC 11.3.0 + Clang 14.0.0-1ubuntu1 ext4 1920x1080 OpenBenchmarking.org Kernel Details - Transparent Huge Pages: madvise Compiler Details - --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details - Scaling Governor: intel_pstate performance (EPP: performance) - CPU Microcode: 0x2b0000c0 Python Details - Python 3.10.6 Security Details - itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected
OpenVINO AMX Benchmarks openvino: Face Detection FP16-INT8 - CPU openvino: Face Detection FP16-INT8 - CPU openvino: Weld Porosity Detection FP16-INT8 - CPU openvino: Weld Porosity Detection FP16-INT8 - CPU openvino: Person Vehicle Bike Detection FP16 - CPU openvino: Person Vehicle Bike Detection FP16 - CPU openvino: Machine Translation EN To DE FP16 - CPU openvino: Machine Translation EN To DE FP16 - CPU AVX512_CORE_AMX AVX512_CORE_FP16 AVX512_CORE_BF16 AVX512_CORE_VNNI AVX512_CORE 282.25 424.16 23066.63 5.18 4778.34 25.09 724.10 82.79 146.08 409.81 8897.10 13.46 2730.07 21.95 321.28 186.53 146.06 409.84 8918.62 13.43 2767.16 21.66 314.61 190.49 146.02 409.95 8909.05 13.44 1821.09 32.93 387.62 154.55 63.21 942.79 5835.39 20.54 1820.27 32.94 378.72 158.26 OpenBenchmarking.org
OpenVINO Model: Face Detection FP16-INT8 - Device: CPU OpenBenchmarking.org FPS, More Is Better OpenVINO 2022.3 Model: Face Detection FP16-INT8 - Device: CPU AVX512_CORE AVX512_CORE_AMX AVX512_CORE_BF16 AVX512_CORE_FP16 AVX512_CORE_VNNI 60 120 180 240 300 SE +/- 0.03, N = 3 SE +/- 2.56, N = 3 SE +/- 0.22, N = 3 SE +/- 0.29, N = 3 SE +/- 0.30, N = 3 63.21 282.25 146.06 146.08 146.02 1. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl
OpenVINO Model: Face Detection FP16-INT8 - Device: CPU OpenBenchmarking.org ms, Fewer Is Better OpenVINO 2022.3 Model: Face Detection FP16-INT8 - Device: CPU AVX512_CORE AVX512_CORE_AMX AVX512_CORE_BF16 AVX512_CORE_FP16 AVX512_CORE_VNNI 200 400 600 800 1000 SE +/- 0.30, N = 3 SE +/- 3.75, N = 3 SE +/- 0.65, N = 3 SE +/- 0.81, N = 3 SE +/- 0.81, N = 3 942.79 424.16 409.84 409.81 409.95 MIN: 383.12 / MAX: 1157.46 MIN: 153.43 / MAX: 1952.95 MIN: 208.02 / MAX: 643.02 MIN: 203.37 / MAX: 909.92 MIN: 216.25 / MAX: 615.53 1. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl
OpenVINO CPU Peak Freq (Highest CPU Core Frequency) Monitor Min Avg Max AVX512_CORE 1900 2649 3515 AVX512_CORE_AMX 1900 2561 3507 AVX512_CORE_BF16 1900 2401 3505 AVX512_CORE_FP16 1231 2374 3508 AVX512_CORE_VNNI 1126 2393 3512 OpenBenchmarking.org Megahertz, More Is Better OpenVINO 2022.3 CPU Peak Freq (Highest CPU Core Frequency) Monitor 1000 2000 3000 4000 5000
OpenVINO CPU Power Consumption Monitor Min Avg Max AVX512_CORE 214 643 759 AVX512_CORE_AMX 208 616 761 AVX512_CORE_BF16 104 641 758 AVX512_CORE_FP16 210 641 748 AVX512_CORE_VNNI 201 640 752 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2022.3 CPU Power Consumption Monitor 200 400 600 800 1000
OpenVINO CPU Temperature Monitor Min Avg Max AVX512_CORE 33.0 52.1 57.0 AVX512_CORE_AMX 30.0 48.0 55.0 AVX512_CORE_BF16 31.0 51.1 56.0 AVX512_CORE_FP16 34.0 52.3 57.0 AVX512_CORE_VNNI 35.0 52.6 59.0 OpenBenchmarking.org Celsius, Fewer Is Better OpenVINO 2022.3 CPU Temperature Monitor 16 32 48 64 80
OpenVINO Model: Weld Porosity Detection FP16-INT8 - Device: CPU OpenBenchmarking.org FPS, More Is Better OpenVINO 2022.3 Model: Weld Porosity Detection FP16-INT8 - Device: CPU AVX512_CORE AVX512_CORE_AMX AVX512_CORE_BF16 AVX512_CORE_FP16 AVX512_CORE_VNNI 5K 10K 15K 20K 25K SE +/- 78.19, N = 3 SE +/- 92.31, N = 3 SE +/- 85.46, N = 3 SE +/- 24.48, N = 3 SE +/- 52.22, N = 3 5835.39 23066.63 8918.62 8897.10 8909.05 1. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl
OpenVINO Model: Weld Porosity Detection FP16-INT8 - Device: CPU OpenBenchmarking.org ms, Fewer Is Better OpenVINO 2022.3 Model: Weld Porosity Detection FP16-INT8 - Device: CPU AVX512_CORE AVX512_CORE_AMX AVX512_CORE_BF16 AVX512_CORE_FP16 AVX512_CORE_VNNI 5 10 15 20 25 SE +/- 0.28, N = 3 SE +/- 0.02, N = 3 SE +/- 0.13, N = 3 SE +/- 0.04, N = 3 SE +/- 0.08, N = 3 20.54 5.18 13.43 13.46 13.44 MIN: 9.17 / MAX: 114 MIN: 1.43 / MAX: 102.62 MIN: 3.72 / MAX: 102.63 MIN: 3.82 / MAX: 88.11 MIN: 3.47 / MAX: 112.16 1. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl
OpenVINO CPU Peak Freq (Highest CPU Core Frequency) Monitor Min Avg Max AVX512_CORE 1900 2674 3503 AVX512_CORE_AMX 1900 2860 3501 AVX512_CORE_BF16 1900 2891 3512 AVX512_CORE_FP16 1900 2928 3512 AVX512_CORE_VNNI 1900 2903 3511 OpenBenchmarking.org Megahertz, More Is Better OpenVINO 2022.3 CPU Peak Freq (Highest CPU Core Frequency) Monitor 1000 2000 3000 4000 5000
OpenVINO CPU Power Consumption Monitor Min Avg Max AVX512_CORE 212 668 759 AVX512_CORE_AMX 217 663 757 AVX512_CORE_BF16 210 667 769 AVX512_CORE_FP16 210 665 756 AVX512_CORE_VNNI 211 665 760 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2022.3 CPU Power Consumption Monitor 200 400 600 800 1000
OpenVINO CPU Temperature Monitor Min Avg Max AVX512_CORE 39.0 54.9 58.0 AVX512_CORE_AMX 39.0 52.2 55.0 AVX512_CORE_BF16 38.0 54.6 58.0 AVX512_CORE_FP16 40.0 55.2 58.0 AVX512_CORE_VNNI 38.0 54.8 58.0 OpenBenchmarking.org Celsius, Fewer Is Better OpenVINO 2022.3 CPU Temperature Monitor 16 32 48 64 80
OpenVINO Model: Person Vehicle Bike Detection FP16 - Device: CPU OpenBenchmarking.org FPS, More Is Better OpenVINO 2022.3 Model: Person Vehicle Bike Detection FP16 - Device: CPU AVX512_CORE AVX512_CORE_AMX AVX512_CORE_BF16 AVX512_CORE_FP16 AVX512_CORE_VNNI 1000 2000 3000 4000 5000 SE +/- 8.67, N = 3 SE +/- 14.65, N = 3 SE +/- 16.62, N = 3 SE +/- 11.17, N = 3 SE +/- 7.34, N = 3 1820.27 4778.34 2767.16 2730.07 1821.09 1. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl
OpenVINO Model: Person Vehicle Bike Detection FP16 - Device: CPU OpenBenchmarking.org ms, Fewer Is Better OpenVINO 2022.3 Model: Person Vehicle Bike Detection FP16 - Device: CPU AVX512_CORE AVX512_CORE_AMX AVX512_CORE_BF16 AVX512_CORE_FP16 AVX512_CORE_VNNI 8 16 24 32 40 SE +/- 0.16, N = 3 SE +/- 0.08, N = 3 SE +/- 0.13, N = 3 SE +/- 0.09, N = 3 SE +/- 0.13, N = 3 32.94 25.09 21.66 21.95 32.93 MIN: 12.45 / MAX: 111.25 MIN: 11.66 / MAX: 132.09 MIN: 13.42 / MAX: 101.56 MIN: 11.62 / MAX: 120.04 MIN: 12.7 / MAX: 103.82 1. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl
OpenVINO CPU Peak Freq (Highest CPU Core Frequency) Monitor Min Avg Max AVX512_CORE 1900 2910 3513 AVX512_CORE_AMX 1900 2965 3501 AVX512_CORE_BF16 1900 2755 3513 AVX512_CORE_FP16 1900 2739 3511 AVX512_CORE_VNNI 1900 2934 3508 OpenBenchmarking.org Megahertz, More Is Better OpenVINO 2022.3 CPU Peak Freq (Highest CPU Core Frequency) Monitor 1000 2000 3000 4000 5000
OpenVINO CPU Power Consumption Monitor Min Avg Max AVX512_CORE 212 655 749 AVX512_CORE_AMX 213 640 755 AVX512_CORE_BF16 210 652 754 AVX512_CORE_FP16 132 653 756 AVX512_CORE_VNNI 210 655 748 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2022.3 CPU Power Consumption Monitor 200 400 600 800 1000
OpenVINO CPU Temperature Monitor Min Avg Max AVX512_CORE 40.0 52.1 55.0 AVX512_CORE_AMX 39.0 51.9 55.0 AVX512_CORE_BF16 39.0 52.5 57.0 AVX512_CORE_FP16 39.0 52.8 57.0 AVX512_CORE_VNNI 39.0 52.0 55.0 OpenBenchmarking.org Celsius, Fewer Is Better OpenVINO 2022.3 CPU Temperature Monitor 16 32 48 64 80
OpenVINO Model: Machine Translation EN To DE FP16 - Device: CPU OpenBenchmarking.org FPS, More Is Better OpenVINO 2022.3 Model: Machine Translation EN To DE FP16 - Device: CPU AVX512_CORE AVX512_CORE_AMX AVX512_CORE_BF16 AVX512_CORE_FP16 AVX512_CORE_VNNI 160 320 480 640 800 SE +/- 5.28, N = 3 SE +/- 5.21, N = 3 SE +/- 3.03, N = 3 SE +/- 0.77, N = 3 SE +/- 2.49, N = 3 378.72 724.10 314.61 321.28 387.62 1. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl
OpenVINO Model: Machine Translation EN To DE FP16 - Device: CPU OpenBenchmarking.org ms, Fewer Is Better OpenVINO 2022.3 Model: Machine Translation EN To DE FP16 - Device: CPU AVX512_CORE AVX512_CORE_AMX AVX512_CORE_BF16 AVX512_CORE_FP16 AVX512_CORE_VNNI 40 80 120 160 200 SE +/- 2.22, N = 3 SE +/- 0.60, N = 3 SE +/- 1.85, N = 3 SE +/- 0.45, N = 3 SE +/- 1.04, N = 3 158.26 82.79 190.49 186.53 154.55 MIN: 62.25 / MAX: 1819.89 MIN: 31.64 / MAX: 1588.97 MIN: 117.21 / MAX: 1606.21 MIN: 92.82 / MAX: 808.7 MIN: 90.51 / MAX: 1925.11 1. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl
OpenVINO CPU Peak Freq (Highest CPU Core Frequency) Monitor Min Avg Max AVX512_CORE 1900 2590 3518 AVX512_CORE_AMX 1900 2792 3516 AVX512_CORE_BF16 1900 2710 3512 AVX512_CORE_FP16 1900 2656 3508 AVX512_CORE_VNNI 1900 2568 3509 OpenBenchmarking.org Megahertz, More Is Better OpenVINO 2022.3 CPU Peak Freq (Highest CPU Core Frequency) Monitor 1000 2000 3000 4000 5000
OpenVINO CPU Power Consumption Monitor Min Avg Max AVX512_CORE 211 648 753 AVX512_CORE_AMX 212 644 742 AVX512_CORE_BF16 209 650 756 AVX512_CORE_FP16 211 652 756 AVX512_CORE_VNNI 208 648 751 OpenBenchmarking.org Watts, Fewer Is Better OpenVINO 2022.3 CPU Power Consumption Monitor 200 400 600 800 1000
OpenVINO CPU Temperature Monitor Min Avg Max AVX512_CORE 40.0 52.4 56.0 AVX512_CORE_AMX 39.0 51.5 55.0 AVX512_CORE_BF16 39.0 52.6 57.0 AVX512_CORE_FP16 38.0 52.6 56.0 AVX512_CORE_VNNI 39.0 52.4 56.0 OpenBenchmarking.org Celsius, Fewer Is Better OpenVINO 2022.3 CPU Temperature Monitor 16 32 48 64 80
Phoronix Test Suite v10.8.4