OpenVINO AMX Benchmarks AMD EPYC Genoa Benchmarks by Michael Larabel for a future article. Xeon Platinum 8490H 2P: Processor: 2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS), Chipset: Intel Device 1bce, Memory: 16 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG, Disk: 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96, Graphics: ASPEED, Monitor: VGA HDMI, Network: 4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-T OS: Ubuntu 22.04, Kernel: 6.1.4-060104-generic (x86_64), Desktop: GNOME Shell 42.2, Display Server: X Server 1.21.1.3, Vulkan: 1.2.204, Compiler: GCC 11.3.0 + Clang 14.0.0-1ubuntu1, File-System: ext4, Screen Resolution: 1920x1080 Xeon Platinum 8490H 2P - No AMX: Processor: 2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS), Chipset: Intel Device 1bce, Memory: 16 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG, Disk: 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96, Graphics: ASPEED, Monitor: VGA HDMI, Network: 4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-T OS: Ubuntu 22.04, Kernel: 6.1.4-060104-generic (x86_64), Desktop: GNOME Shell 42.2, Display Server: X Server 1.21.1.3, Vulkan: 1.2.204, Compiler: GCC 11.3.0 + Clang 14.0.0-1ubuntu1, File-System: ext4, Screen Resolution: 1920x1080 EPYC 9654 2P: Processor: 2 x AMD EPYC 9654 96-Core @ 2.40GHz (192 Cores / 384 Threads), Motherboard: AMD Titanite_4G (RTI1002E BIOS), Chipset: AMD Device 14a4, Memory: 24 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG, Disk: 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007, Graphics: ASPEED, Monitor: VGA HDMI, Network: Broadcom NetXtreme BCM5720 PCIe OS: Ubuntu 22.04, Kernel: 6.1.4-060104-generic (x86_64), Desktop: GNOME Shell 42.2, Display Server: X Server 1.21.1.3, Vulkan: 1.2.204, Compiler: GCC 11.3.0 + Clang 14.0.0-1ubuntu1, File-System: ext4, Screen Resolution: 1920x1080 OpenVINO 2022.3 Model: Face Detection FP16-INT8 - Device: CPU FPS > Higher Is Better Xeon Platinum 8490H 2P .......... 282.25 |===================================== Xeon Platinum 8490H 2P - No AMX . 146.08 |=================== EPYC 9654 2P .................... 188.64 |========================= OpenVINO 2022.3 Model: Face Detection FP16-INT8 - Device: CPU ms < Lower Is Better Xeon Platinum 8490H 2P .......... 424.16 |=============================== Xeon Platinum 8490H 2P - No AMX . 409.81 |============================== EPYC 9654 2P .................... 507.25 |===================================== OpenVINO 2022.3 Model: Weld Porosity Detection FP16-INT8 - Device: CPU FPS > Higher Is Better Xeon Platinum 8490H 2P .......... 23066.63 |=================================== Xeon Platinum 8490H 2P - No AMX . 8897.10 |============= EPYC 9654 2P .................... 18678.16 |============================ OpenVINO 2022.3 Model: Weld Porosity Detection FP16-INT8 - Device: CPU ms < Lower Is Better Xeon Platinum 8490H 2P .......... 5.18 |=============== Xeon Platinum 8490H 2P - No AMX . 13.46 |====================================== EPYC 9654 2P .................... 10.26 |============================= OpenVINO 2022.3 Model: Person Vehicle Bike Detection FP16 - Device: CPU FPS > Higher Is Better Xeon Platinum 8490H 2P .......... 4778.34 |================================ Xeon Platinum 8490H 2P - No AMX . 2730.07 |================== EPYC 9654 2P .................... 5320.83 |==================================== OpenVINO 2022.3 Model: Person Vehicle Bike Detection FP16 - Device: CPU ms < Lower Is Better Xeon Platinum 8490H 2P .......... 25.09 |====================================== Xeon Platinum 8490H 2P - No AMX . 21.95 |================================= EPYC 9654 2P .................... 18.03 |=========================== OpenVINO 2022.3 Model: Machine Translation EN To DE FP16 - Device: CPU FPS > Higher Is Better Xeon Platinum 8490H 2P .......... 724.10 |=========================== Xeon Platinum 8490H 2P - No AMX . 321.28 |============ EPYC 9654 2P .................... 989.16 |===================================== OpenVINO 2022.3 Model: Machine Translation EN To DE FP16 - Device: CPU ms < Lower Is Better Xeon Platinum 8490H 2P .......... 82.79 |================ Xeon Platinum 8490H 2P - No AMX . 186.53 |===================================== EPYC 9654 2P .................... 97.00 |=================== OpenVINO 2022.3 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better Xeon Platinum 8490H 2P .......... MIN: 1900 AVG: 2561 MAX: 3507 Xeon Platinum 8490H 2P - No AMX . MIN: 1231 AVG: 2374 MAX: 3508 EPYC 9654 2P .................... MIN: 2400 AVG: 2990 MAX: 3702 OpenVINO 2022.3 CPU Power Consumption Monitor Watts < Lower Is Better Xeon Platinum 8490H 2P .......... MIN: 208 AVG: 616 MAX: 761 Xeon Platinum 8490H 2P - No AMX . MIN: 210 AVG: 641 MAX: 748 EPYC 9654 2P .................... MIN: 54 AVG: 602 MAX: 717 OpenVINO 2022.3 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better Xeon Platinum 8490H 2P .......... MIN: 1900 AVG: 2860 MAX: 3501 Xeon Platinum 8490H 2P - No AMX . MIN: 1900 AVG: 2928 MAX: 3512 EPYC 9654 2P .................... MIN: 2400 AVG: 2809 MAX: 3702 OpenVINO 2022.3 CPU Power Consumption Monitor Watts < Lower Is Better Xeon Platinum 8490H 2P .......... MIN: 217 AVG: 663 MAX: 757 Xeon Platinum 8490H 2P - No AMX . MIN: 210 AVG: 665 MAX: 756 EPYC 9654 2P .................... MIN: 59 AVG: 652 MAX: 717 OpenVINO 2022.3 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better Xeon Platinum 8490H 2P .......... MIN: 1900 AVG: 2965 MAX: 3501 Xeon Platinum 8490H 2P - No AMX . MIN: 1900 AVG: 2739 MAX: 3511 EPYC 9654 2P .................... MIN: 2400 AVG: 3493 MAX: 3763 OpenVINO 2022.3 CPU Power Consumption Monitor Watts < Lower Is Better Xeon Platinum 8490H 2P .......... MIN: 213 AVG: 640 MAX: 755 Xeon Platinum 8490H 2P - No AMX . MIN: 132 AVG: 653 MAX: 756 EPYC 9654 2P .................... MIN: 55 AVG: 567 MAX: 643 OpenVINO 2022.3 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better Xeon Platinum 8490H 2P .......... MIN: 1900 AVG: 2792 MAX: 3516 Xeon Platinum 8490H 2P - No AMX . MIN: 1900 AVG: 2656 MAX: 3508 EPYC 9654 2P .................... MIN: 2400 AVG: 2965 MAX: 3700 OpenVINO 2022.3 CPU Power Consumption Monitor Watts < Lower Is Better Xeon Platinum 8490H 2P .......... MIN: 212 AVG: 644 MAX: 742 Xeon Platinum 8490H 2P - No AMX . MIN: 211 AVG: 652 MAX: 756 EPYC 9654 2P .................... MIN: 55 AVG: 650 MAX: 740 CPU Peak Freq (Highest CPU Core Frequency) Monitor Phoronix Test Suite System Monitoring Megahertz Xeon Platinum 8490H 2P .......... MIN: 1900 AVG: 2744 MAX: 3516 Xeon Platinum 8490H 2P - No AMX . MIN: 1000 AVG: 2604 MAX: 3512 EPYC 9654 2P .................... MIN: 2400 AVG: 3129 MAX: 3763 CPU Power Consumption Monitor Phoronix Test Suite System Monitoring Watts Xeon Platinum 8490H 2P .......... MIN: 208 AVG: 624 MAX: 761 Xeon Platinum 8490H 2P - No AMX . MIN: 123 AVG: 646 MAX: 764 EPYC 9654 2P .................... MIN: 53 AVG: 596 MAX: 740