OpenVINO AMX Benchmarks Benchmarks by Michael Larabel for a future article. AVX512_CORE_AMX: Processor: 2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS), Chipset: Intel Device 1bce, Memory: 16 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG, Disk: 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96, Graphics: ASPEED, Monitor: VGA HDMI, Network: 4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-T OS: Ubuntu 22.04, Kernel: 6.1.4-060104-generic (x86_64), Desktop: GNOME Shell 42.2, Display Server: X Server 1.21.1.3, Vulkan: 1.2.204, Compiler: GCC 11.3.0 + Clang 14.0.0-1ubuntu1, File-System: ext4, Screen Resolution: 1920x1080 AVX512_CORE_FP16: Processor: 2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS), Chipset: Intel Device 1bce, Memory: 16 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG, Disk: 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96, Graphics: ASPEED, Monitor: VGA HDMI, Network: 4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-T OS: Ubuntu 22.04, Kernel: 6.1.4-060104-generic (x86_64), Desktop: GNOME Shell 42.2, Display Server: X Server 1.21.1.3, Vulkan: 1.2.204, Compiler: GCC 11.3.0 + Clang 14.0.0-1ubuntu1, File-System: ext4, Screen Resolution: 1920x1080 AVX512_CORE_BF16: Processor: 2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS), Chipset: Intel Device 1bce, Memory: 16 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG, Disk: 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96, Graphics: ASPEED, Monitor: VGA HDMI, Network: 4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-T OS: Ubuntu 22.04, Kernel: 6.1.4-060104-generic (x86_64), Desktop: GNOME Shell 42.2, Display Server: X Server 1.21.1.3, Vulkan: 1.2.204, Compiler: GCC 11.3.0 + Clang 14.0.0-1ubuntu1, File-System: ext4, Screen Resolution: 1920x1080 AVX512_CORE_VNNI: Processor: 2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS), Chipset: Intel Device 1bce, Memory: 16 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG, Disk: 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96, Graphics: ASPEED, Monitor: VGA HDMI, Network: 4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-T OS: Ubuntu 22.04, Kernel: 6.1.4-060104-generic (x86_64), Desktop: GNOME Shell 42.2, Display Server: X Server 1.21.1.3, Vulkan: 1.2.204, Compiler: GCC 11.3.0 + Clang 14.0.0-1ubuntu1, File-System: ext4, Screen Resolution: 1920x1080 AVX512_CORE: Processor: 2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS), Chipset: Intel Device 1bce, Memory: 16 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG, Disk: 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96, Graphics: ASPEED, Monitor: VGA HDMI, Network: 4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-T OS: Ubuntu 22.04, Kernel: 6.1.4-060104-generic (x86_64), Desktop: GNOME Shell 42.2, Display Server: X Server 1.21.1.3, Vulkan: 1.2.204, Compiler: GCC 11.3.0 + Clang 14.0.0-1ubuntu1, File-System: ext4, Screen Resolution: 1920x1080 OpenVINO 2022.3 Model: Face Detection FP16-INT8 - Device: CPU FPS > Higher Is Better AVX512_CORE_AMX .. 282.25 |==================================================== AVX512_CORE_FP16 . 146.08 |=========================== AVX512_CORE_BF16 . 146.06 |=========================== AVX512_CORE_VNNI . 146.02 |=========================== AVX512_CORE ...... 63.21 |============ OpenVINO 2022.3 Model: Face Detection FP16-INT8 - Device: CPU ms < Lower Is Better AVX512_CORE_AMX .. 424.16 |======================= AVX512_CORE_FP16 . 409.81 |======================= AVX512_CORE_BF16 . 409.84 |======================= AVX512_CORE_VNNI . 409.95 |======================= AVX512_CORE ...... 942.79 |==================================================== OpenVINO 2022.3 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better AVX512_CORE_AMX .. MIN: 1900 AVG: 2561 MAX: 3507 AVX512_CORE_FP16 . MIN: 1231 AVG: 2374 MAX: 3508 AVX512_CORE_BF16 . MIN: 1900 AVG: 2401 MAX: 3505 AVX512_CORE_VNNI . MIN: 1126 AVG: 2393 MAX: 3512 AVX512_CORE ...... MIN: 1900 AVG: 2649 MAX: 3515 OpenVINO 2022.3 CPU Power Consumption Monitor Watts < Lower Is Better AVX512_CORE_AMX .. MIN: 208 AVG: 616 MAX: 761 AVX512_CORE_FP16 . MIN: 210 AVG: 641 MAX: 748 AVX512_CORE_BF16 . MIN: 104 AVG: 641 MAX: 758 AVX512_CORE_VNNI . MIN: 201 AVG: 640 MAX: 752 AVX512_CORE ...... MIN: 214 AVG: 643 MAX: 759 OpenVINO 2022.3 CPU Temperature Monitor Celsius < Lower Is Better AVX512_CORE_AMX .. MIN: 30.0 AVG: 48.0 MAX: 55.0 AVX512_CORE_FP16 . MIN: 34.0 AVG: 52.3 MAX: 57.0 AVX512_CORE_BF16 . MIN: 31.0 AVG: 51.1 MAX: 56.0 AVX512_CORE_VNNI . MIN: 35.0 AVG: 52.6 MAX: 59.0 AVX512_CORE ...... MIN: 33.0 AVG: 52.1 MAX: 57.0 OpenVINO 2022.3 Model: Weld Porosity Detection FP16-INT8 - Device: CPU FPS > Higher Is Better AVX512_CORE_AMX .. 23066.63 |================================================== AVX512_CORE_FP16 . 8897.10 |=================== AVX512_CORE_BF16 . 8918.62 |=================== AVX512_CORE_VNNI . 8909.05 |=================== AVX512_CORE ...... 5835.39 |============= OpenVINO 2022.3 Model: Weld Porosity Detection FP16-INT8 - Device: CPU ms < Lower Is Better AVX512_CORE_AMX .. 5.18 |============= AVX512_CORE_FP16 . 13.46 |=================================== AVX512_CORE_BF16 . 13.43 |=================================== AVX512_CORE_VNNI . 13.44 |=================================== AVX512_CORE ...... 20.54 |===================================================== OpenVINO 2022.3 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better AVX512_CORE_AMX .. MIN: 1900 AVG: 2860 MAX: 3501 AVX512_CORE_FP16 . MIN: 1900 AVG: 2928 MAX: 3512 AVX512_CORE_BF16 . MIN: 1900 AVG: 2891 MAX: 3512 AVX512_CORE_VNNI . MIN: 1900 AVG: 2903 MAX: 3511 AVX512_CORE ...... MIN: 1900 AVG: 2674 MAX: 3503 OpenVINO 2022.3 CPU Power Consumption Monitor Watts < Lower Is Better AVX512_CORE_AMX .. MIN: 217 AVG: 663 MAX: 757 AVX512_CORE_FP16 . MIN: 210 AVG: 665 MAX: 756 AVX512_CORE_BF16 . MIN: 210 AVG: 667 MAX: 769 AVX512_CORE_VNNI . MIN: 211 AVG: 665 MAX: 760 AVX512_CORE ...... MIN: 212 AVG: 668 MAX: 759 OpenVINO 2022.3 CPU Temperature Monitor Celsius < Lower Is Better AVX512_CORE_AMX .. MIN: 39.0 AVG: 52.2 MAX: 55.0 AVX512_CORE_FP16 . MIN: 40.0 AVG: 55.2 MAX: 58.0 AVX512_CORE_BF16 . MIN: 38.0 AVG: 54.6 MAX: 58.0 AVX512_CORE_VNNI . MIN: 38.0 AVG: 54.8 MAX: 58.0 AVX512_CORE ...... MIN: 39.0 AVG: 54.9 MAX: 58.0 OpenVINO 2022.3 Model: Person Vehicle Bike Detection FP16 - Device: CPU FPS > Higher Is Better AVX512_CORE_AMX .. 4778.34 |=================================================== AVX512_CORE_FP16 . 2730.07 |============================= AVX512_CORE_BF16 . 2767.16 |============================== AVX512_CORE_VNNI . 1821.09 |=================== AVX512_CORE ...... 1820.27 |=================== OpenVINO 2022.3 Model: Person Vehicle Bike Detection FP16 - Device: CPU ms < Lower Is Better AVX512_CORE_AMX .. 25.09 |======================================== AVX512_CORE_FP16 . 21.95 |=================================== AVX512_CORE_BF16 . 21.66 |=================================== AVX512_CORE_VNNI . 32.93 |===================================================== AVX512_CORE ...... 32.94 |===================================================== OpenVINO 2022.3 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better AVX512_CORE_AMX .. MIN: 1900 AVG: 2965 MAX: 3501 AVX512_CORE_FP16 . MIN: 1900 AVG: 2739 MAX: 3511 AVX512_CORE_BF16 . MIN: 1900 AVG: 2755 MAX: 3513 AVX512_CORE_VNNI . MIN: 1900 AVG: 2934 MAX: 3508 AVX512_CORE ...... MIN: 1900 AVG: 2910 MAX: 3513 OpenVINO 2022.3 CPU Power Consumption Monitor Watts < Lower Is Better AVX512_CORE_AMX .. MIN: 213 AVG: 640 MAX: 755 AVX512_CORE_FP16 . MIN: 132 AVG: 653 MAX: 756 AVX512_CORE_BF16 . MIN: 210 AVG: 652 MAX: 754 AVX512_CORE_VNNI . MIN: 210 AVG: 655 MAX: 748 AVX512_CORE ...... MIN: 212 AVG: 655 MAX: 749 OpenVINO 2022.3 CPU Temperature Monitor Celsius < Lower Is Better AVX512_CORE_AMX .. MIN: 39.0 AVG: 51.9 MAX: 55.0 AVX512_CORE_FP16 . MIN: 39.0 AVG: 52.8 MAX: 57.0 AVX512_CORE_BF16 . MIN: 39.0 AVG: 52.5 MAX: 57.0 AVX512_CORE_VNNI . MIN: 39.0 AVG: 52.0 MAX: 55.0 AVX512_CORE ...... MIN: 40.0 AVG: 52.1 MAX: 55.0 OpenVINO 2022.3 Model: Machine Translation EN To DE FP16 - Device: CPU FPS > Higher Is Better AVX512_CORE_AMX .. 724.10 |==================================================== AVX512_CORE_FP16 . 321.28 |======================= AVX512_CORE_BF16 . 314.61 |======================= AVX512_CORE_VNNI . 387.62 |============================ AVX512_CORE ...... 378.72 |=========================== OpenVINO 2022.3 Model: Machine Translation EN To DE FP16 - Device: CPU ms < Lower Is Better AVX512_CORE_AMX .. 82.79 |======================= AVX512_CORE_FP16 . 186.53 |=================================================== AVX512_CORE_BF16 . 190.49 |==================================================== AVX512_CORE_VNNI . 154.55 |========================================== AVX512_CORE ...... 158.26 |=========================================== OpenVINO 2022.3 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better AVX512_CORE_AMX .. MIN: 1900 AVG: 2792 MAX: 3516 AVX512_CORE_FP16 . MIN: 1900 AVG: 2656 MAX: 3508 AVX512_CORE_BF16 . MIN: 1900 AVG: 2710 MAX: 3512 AVX512_CORE_VNNI . MIN: 1900 AVG: 2568 MAX: 3509 AVX512_CORE ...... MIN: 1900 AVG: 2590 MAX: 3518 OpenVINO 2022.3 CPU Power Consumption Monitor Watts < Lower Is Better AVX512_CORE_AMX .. MIN: 212 AVG: 644 MAX: 742 AVX512_CORE_FP16 . MIN: 211 AVG: 652 MAX: 756 AVX512_CORE_BF16 . MIN: 209 AVG: 650 MAX: 756 AVX512_CORE_VNNI . MIN: 208 AVG: 648 MAX: 751 AVX512_CORE ...... MIN: 211 AVG: 648 MAX: 753 OpenVINO 2022.3 CPU Temperature Monitor Celsius < Lower Is Better AVX512_CORE_AMX .. MIN: 39.0 AVG: 51.5 MAX: 55.0 AVX512_CORE_FP16 . MIN: 38.0 AVG: 52.6 MAX: 56.0 AVX512_CORE_BF16 . MIN: 39.0 AVG: 52.6 MAX: 57.0 AVX512_CORE_VNNI . MIN: 39.0 AVG: 52.4 MAX: 56.0 AVX512_CORE ...... MIN: 40.0 AVG: 52.4 MAX: 56.0