OpenVINO AMX Benchmarks

Benchmarks by Michael Larabel for a future article.

Compare your own system(s) to this result file with the Phoronix Test Suite by running the command: phoronix-test-suite benchmark 2301160-NE-OPENVINOA07

Jump To Table - Results

View

Do Not Show Noisy Results
Do Not Show Results With Incomplete Data
Do Not Show Results With Little Change/Spread
List Notable Results

Statistics

Show Overall Harmonic Mean(s)
Show Overall Geometric Mean
Show Wins / Losses Counts (Pie Chart)
Normalize Results
Remove Outliers Before Calculating Averages

Graph Settings

Force Line Graphs Where Applicable
Convert To Scalar Where Applicable
Prefer Vertical Bar Graphs
No Box Plots
On Line Graphs With Missing Data, Connect The Line Gaps

Multi-Way Comparison

Condense Multi-Option Tests Into Single Result Graphs

Table

Show Detailed System Result Table

Run Management

Only show results where is faster than

Only show results matching title/arguments (delimit multiple options with a comma):

Do not show results matching title/arguments (delimit multiple options with a comma):

OpenVINO AMX Benchmarks Benchmarks by Michael Larabel for a future article. AVX512_CORE_AMX: Processor: 2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS), Chipset: Intel Device 1bce, Memory: 16 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG, Disk: 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96, Graphics: ASPEED, Monitor: VGA HDMI, Network: 4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-T OS: Ubuntu 22.04, Kernel: 6.1.4-060104-generic (x86_64), Desktop: GNOME Shell 42.2, Display Server: X Server 1.21.1.3, Vulkan: 1.2.204, Compiler: GCC 11.3.0 + Clang 14.0.0-1ubuntu1, File-System: ext4, Screen Resolution: 1920x1080 AVX512_CORE_FP16: Processor: 2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS), Chipset: Intel Device 1bce, Memory: 16 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG, Disk: 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96, Graphics: ASPEED, Monitor: VGA HDMI, Network: 4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-T OS: Ubuntu 22.04, Kernel: 6.1.4-060104-generic (x86_64), Desktop: GNOME Shell 42.2, Display Server: X Server 1.21.1.3, Vulkan: 1.2.204, Compiler: GCC 11.3.0 + Clang 14.0.0-1ubuntu1, File-System: ext4, Screen Resolution: 1920x1080 AVX512_CORE_BF16: Processor: 2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS), Chipset: Intel Device 1bce, Memory: 16 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG, Disk: 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96, Graphics: ASPEED, Monitor: VGA HDMI, Network: 4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-T OS: Ubuntu 22.04, Kernel: 6.1.4-060104-generic (x86_64), Desktop: GNOME Shell 42.2, Display Server: X Server 1.21.1.3, Vulkan: 1.2.204, Compiler: GCC 11.3.0 + Clang 14.0.0-1ubuntu1, File-System: ext4, Screen Resolution: 1920x1080 AVX512_CORE_VNNI: Processor: 2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS), Chipset: Intel Device 1bce, Memory: 16 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG, Disk: 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96, Graphics: ASPEED, Monitor: VGA HDMI, Network: 4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-T OS: Ubuntu 22.04, Kernel: 6.1.4-060104-generic (x86_64), Desktop: GNOME Shell 42.2, Display Server: X Server 1.21.1.3, Vulkan: 1.2.204, Compiler: GCC 11.3.0 + Clang 14.0.0-1ubuntu1, File-System: ext4, Screen Resolution: 1920x1080 AVX512_CORE: Processor: 2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS), Chipset: Intel Device 1bce, Memory: 16 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG, Disk: 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96, Graphics: ASPEED, Monitor: VGA HDMI, Network: 4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-T OS: Ubuntu 22.04, Kernel: 6.1.4-060104-generic (x86_64), Desktop: GNOME Shell 42.2, Display Server: X Server 1.21.1.3, Vulkan: 1.2.204, Compiler: GCC 11.3.0 + Clang 14.0.0-1ubuntu1, File-System: ext4, Screen Resolution: 1920x1080 OpenVINO 2022.3 Model: Face Detection FP16-INT8 - Device: CPU FPS > Higher Is Better AVX512_CORE_AMX .. 282.25 |==================================================== AVX512_CORE_FP16 . 146.08 |=========================== AVX512_CORE_BF16 . 146.06 |=========================== AVX512_CORE_VNNI . 146.02 |=========================== AVX512_CORE ...... 63.21 |============ OpenVINO 2022.3 Model: Face Detection FP16-INT8 - Device: CPU ms < Lower Is Better AVX512_CORE_AMX .. 424.16 |======================= AVX512_CORE_FP16 . 409.81 |======================= AVX512_CORE_BF16 . 409.84 |======================= AVX512_CORE_VNNI . 409.95 |======================= AVX512_CORE ...... 942.79 |==================================================== OpenVINO 2022.3 Model: Weld Porosity Detection FP16-INT8 - Device: CPU FPS > Higher Is Better AVX512_CORE_AMX .. 23066.63 |================================================== AVX512_CORE_FP16 . 8897.10 |=================== AVX512_CORE_BF16 . 8918.62 |=================== AVX512_CORE_VNNI . 8909.05 |=================== AVX512_CORE ...... 5835.39 |============= OpenVINO 2022.3 Model: Weld Porosity Detection FP16-INT8 - Device: CPU ms < Lower Is Better AVX512_CORE_AMX .. 5.18 |============= AVX512_CORE_FP16 . 13.46 |=================================== AVX512_CORE_BF16 . 13.43 |=================================== AVX512_CORE_VNNI . 13.44 |=================================== AVX512_CORE ...... 20.54 |===================================================== OpenVINO 2022.3 Model: Person Vehicle Bike Detection FP16 - Device: CPU FPS > Higher Is Better AVX512_CORE_AMX .. 4778.34 |=================================================== AVX512_CORE_FP16 . 2730.07 |============================= AVX512_CORE_BF16 . 2767.16 |============================== AVX512_CORE_VNNI . 1821.09 |=================== AVX512_CORE ...... 1820.27 |=================== OpenVINO 2022.3 Model: Person Vehicle Bike Detection FP16 - Device: CPU ms < Lower Is Better AVX512_CORE_AMX .. 25.09 |======================================== AVX512_CORE_FP16 . 21.95 |=================================== AVX512_CORE_BF16 . 21.66 |=================================== AVX512_CORE_VNNI . 32.93 |===================================================== AVX512_CORE ...... 32.94 |===================================================== OpenVINO 2022.3 Model: Machine Translation EN To DE FP16 - Device: CPU FPS > Higher Is Better AVX512_CORE_AMX .. 724.10 |==================================================== AVX512_CORE_FP16 . 321.28 |======================= AVX512_CORE_BF16 . 314.61 |======================= AVX512_CORE_VNNI . 387.62 |============================ AVX512_CORE ...... 378.72 |=========================== OpenVINO 2022.3 Model: Machine Translation EN To DE FP16 - Device: CPU ms < Lower Is Better AVX512_CORE_AMX .. 82.79 |======================= AVX512_CORE_FP16 . 186.53 |=================================================== AVX512_CORE_BF16 . 190.49 |==================================================== AVX512_CORE_VNNI . 154.55 |========================================== AVX512_CORE ...... 158.26 |=========================================== OpenVINO 2022.3 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better AVX512_CORE_AMX .. MIN: 1900 AVG: 2561 MAX: 3507 AVX512_CORE_FP16 . MIN: 1231 AVG: 2374 MAX: 3508 AVX512_CORE_BF16 . MIN: 1900 AVG: 2401 MAX: 3505 AVX512_CORE_VNNI . MIN: 1126 AVG: 2393 MAX: 3512 AVX512_CORE ...... MIN: 1900 AVG: 2649 MAX: 3515 OpenVINO 2022.3 CPU Power Consumption Monitor Watts < Lower Is Better AVX512_CORE_AMX .. MIN: 208 AVG: 616 MAX: 761 AVX512_CORE_FP16 . MIN: 210 AVG: 641 MAX: 748 AVX512_CORE_BF16 . MIN: 104 AVG: 641 MAX: 758 AVX512_CORE_VNNI . MIN: 201 AVG: 640 MAX: 752 AVX512_CORE ...... MIN: 214 AVG: 643 MAX: 759 OpenVINO 2022.3 CPU Temperature Monitor Celsius < Lower Is Better AVX512_CORE_AMX .. MIN: 30.0 AVG: 48.0 MAX: 55.0 AVX512_CORE_FP16 . MIN: 34.0 AVG: 52.3 MAX: 57.0 AVX512_CORE_BF16 . MIN: 31.0 AVG: 51.1 MAX: 56.0 AVX512_CORE_VNNI . MIN: 35.0 AVG: 52.6 MAX: 59.0 AVX512_CORE ...... MIN: 33.0 AVG: 52.1 MAX: 57.0 OpenVINO 2022.3 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better AVX512_CORE_AMX .. MIN: 1900 AVG: 2860 MAX: 3501 AVX512_CORE_FP16 . MIN: 1900 AVG: 2928 MAX: 3512 AVX512_CORE_BF16 . MIN: 1900 AVG: 2891 MAX: 3512 AVX512_CORE_VNNI . MIN: 1900 AVG: 2903 MAX: 3511 AVX512_CORE ...... MIN: 1900 AVG: 2674 MAX: 3503 OpenVINO 2022.3 CPU Power Consumption Monitor Watts < Lower Is Better AVX512_CORE_AMX .. MIN: 217 AVG: 663 MAX: 757 AVX512_CORE_FP16 . MIN: 210 AVG: 665 MAX: 756 AVX512_CORE_BF16 . MIN: 210 AVG: 667 MAX: 769 AVX512_CORE_VNNI . MIN: 211 AVG: 665 MAX: 760 AVX512_CORE ...... MIN: 212 AVG: 668 MAX: 759 OpenVINO 2022.3 CPU Temperature Monitor Celsius < Lower Is Better AVX512_CORE_AMX .. MIN: 39.0 AVG: 52.2 MAX: 55.0 AVX512_CORE_FP16 . MIN: 40.0 AVG: 55.2 MAX: 58.0 AVX512_CORE_BF16 . MIN: 38.0 AVG: 54.6 MAX: 58.0 AVX512_CORE_VNNI . MIN: 38.0 AVG: 54.8 MAX: 58.0 AVX512_CORE ...... MIN: 39.0 AVG: 54.9 MAX: 58.0 OpenVINO 2022.3 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better AVX512_CORE_AMX .. MIN: 1900 AVG: 2965 MAX: 3501 AVX512_CORE_FP16 . MIN: 1900 AVG: 2739 MAX: 3511 AVX512_CORE_BF16 . MIN: 1900 AVG: 2755 MAX: 3513 AVX512_CORE_VNNI . MIN: 1900 AVG: 2934 MAX: 3508 AVX512_CORE ...... MIN: 1900 AVG: 2910 MAX: 3513 OpenVINO 2022.3 CPU Power Consumption Monitor Watts < Lower Is Better AVX512_CORE_AMX .. MIN: 213 AVG: 640 MAX: 755 AVX512_CORE_FP16 . MIN: 132 AVG: 653 MAX: 756 AVX512_CORE_BF16 . MIN: 210 AVG: 652 MAX: 754 AVX512_CORE_VNNI . MIN: 210 AVG: 655 MAX: 748 AVX512_CORE ...... MIN: 212 AVG: 655 MAX: 749 OpenVINO 2022.3 CPU Temperature Monitor Celsius < Lower Is Better AVX512_CORE_AMX .. MIN: 39.0 AVG: 51.9 MAX: 55.0 AVX512_CORE_FP16 . MIN: 39.0 AVG: 52.8 MAX: 57.0 AVX512_CORE_BF16 . MIN: 39.0 AVG: 52.5 MAX: 57.0 AVX512_CORE_VNNI . MIN: 39.0 AVG: 52.0 MAX: 55.0 AVX512_CORE ...... MIN: 40.0 AVG: 52.1 MAX: 55.0 OpenVINO 2022.3 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better AVX512_CORE_AMX .. MIN: 1900 AVG: 2792 MAX: 3516 AVX512_CORE_FP16 . MIN: 1900 AVG: 2656 MAX: 3508 AVX512_CORE_BF16 . MIN: 1900 AVG: 2710 MAX: 3512 AVX512_CORE_VNNI . MIN: 1900 AVG: 2568 MAX: 3509 AVX512_CORE ...... MIN: 1900 AVG: 2590 MAX: 3518 OpenVINO 2022.3 CPU Power Consumption Monitor Watts < Lower Is Better AVX512_CORE_AMX .. MIN: 212 AVG: 644 MAX: 742 AVX512_CORE_FP16 . MIN: 211 AVG: 652 MAX: 756 AVX512_CORE_BF16 . MIN: 209 AVG: 650 MAX: 756 AVX512_CORE_VNNI . MIN: 208 AVG: 648 MAX: 751 AVX512_CORE ...... MIN: 211 AVG: 648 MAX: 753 OpenVINO 2022.3 CPU Temperature Monitor Celsius < Lower Is Better AVX512_CORE_AMX .. MIN: 39.0 AVG: 51.5 MAX: 55.0 AVX512_CORE_FP16 . MIN: 38.0 AVG: 52.6 MAX: 56.0 AVX512_CORE_BF16 . MIN: 39.0 AVG: 52.6 MAX: 57.0 AVX512_CORE_VNNI . MIN: 39.0 AVG: 52.4 MAX: 56.0 AVX512_CORE ...... MIN: 40.0 AVG: 52.4 MAX: 56.0