OONX Sapphire Rapids 2 x Intel Xeon Platinum 8490H testing with a Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS) and ASPEED on Ubuntu 23.04 via the Phoronix Test Suite. ONNX Runtime 1.14: Processor: 2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS), Chipset: Intel Device 1bce, Memory: 1008GB, Disk: 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96, Graphics: ASPEED, Monitor: VGA HDMI, Network: 4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-T OS: Ubuntu 23.04, Kernel: 5.19.0-21-generic (x86_64), Desktop: GNOME Shell 43.2, Display Server: X Server 1.21.1.6, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1080 ONNX Runtime 1.14 - No AMX: Processor: 2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS), Chipset: Intel Device 1bce, Memory: 1008GB, Disk: 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96, Graphics: ASPEED, Monitor: VGA HDMI, Network: 4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-T OS: Ubuntu 23.04, Kernel: 5.19.0-21-generic (x86_64), Desktop: GNOME Shell 43.2, Display Server: X Server 1.21.1.6, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1080 ONNX Runtime 1.14 use_dnnl: Processor: 2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS), Chipset: Intel Device 1bce, Memory: 1008GB, Disk: 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96, Graphics: ASPEED, Monitor: VGA HDMI, Network: 4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-T OS: Ubuntu 23.04, Kernel: 5.19.0-21-generic (x86_64), Desktop: GNOME Shell 43.2, Display Server: X Server 1.21.1.6, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1080 CPU Peak Freq (Highest CPU Core Frequency) Monitor Phoronix Test Suite System Monitoring Megahertz ONNX Runtime 1.14 .......... MIN: 1900 AVG: 3026 MAX: 3666 ONNX Runtime 1.14 use_dnnl . MIN: 1900 AVG: 3073 MAX: 3677 CPU Power Consumption Monitor Phoronix Test Suite System Monitoring Watts ONNX Runtime 1.14 .......... MIN: 122 AVG: 577 MAX: 709 ONNX Runtime 1.14 use_dnnl . MIN: 105 AVG: 571 MAX: 709 CPU Temperature Monitor Phoronix Test Suite System Monitoring Celsius ONNX Runtime 1.14 .......... MIN: 28.0 AVG: 47.6 MAX: 55.0 ONNX Runtime 1.14 use_dnnl . MIN: 32.0 AVG: 48.4 MAX: 55.0 ONNX Runtime 1.14 Model: yolov4 - Device: CPU - Executor: Standard Inferences Per Second > Higher Is Better ONNX Runtime 1.14 .......... 11.54 |=========================================== ONNX Runtime 1.14 - No AMX . 11.27 |========================================== ONNX Runtime 1.14 use_dnnl . 11.20 |========================================== ONNX Runtime 1.14 Model: yolov4 - Device: CPU - Executor: Standard Inference Time Cost (ms) < Lower Is Better ONNX Runtime 1.14 .......... 86.67 |========================================== ONNX Runtime 1.14 - No AMX . 88.93 |=========================================== ONNX Runtime 1.14 use_dnnl . 89.39 |=========================================== ONNX Runtime 1.14 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better ONNX Runtime 1.14 .......... MIN: 1900 AVG: 2919 MAX: 3512 ONNX Runtime 1.14 - No AMX . MIN: 1900 AVG: 2905 MAX: 3511 ONNX Runtime 1.14 use_dnnl . MIN: 1900 AVG: 2911 MAX: 3514 ONNX Runtime 1.14 CPU Power Consumption Monitor Watts < Lower Is Better ONNX Runtime 1.14 .......... MIN: 203 AVG: 611 MAX: 638 ONNX Runtime 1.14 - No AMX . MIN: 201 AVG: 614 MAX: 639 ONNX Runtime 1.14 use_dnnl . MIN: 197 AVG: 613 MAX: 639 ONNX Runtime 1.14 CPU Temperature Monitor Celsius < Lower Is Better ONNX Runtime 1.14 .......... MIN: 28.0 AVG: 46.2 MAX: 50.0 ONNX Runtime 1.14 - No AMX . MIN: 29.0 AVG: 48.2 MAX: 51.0 ONNX Runtime 1.14 use_dnnl . MIN: 33.0 AVG: 48.4 MAX: 51.0 ONNX Runtime 1.14 Model: yolov4 - Device: CPU - Executor: Parallel Inferences Per Second > Higher Is Better ONNX Runtime 1.14 .......... 9.36451 |======================================== ONNX Runtime 1.14 - No AMX . 9.47631 |========================================= ONNX Runtime 1.14 use_dnnl . 9.56208 |========================================= ONNX Runtime 1.14 Model: yolov4 - Device: CPU - Executor: Parallel Inference Time Cost (ms) < Lower Is Better ONNX Runtime 1.14 .......... 106.80 |========================================== ONNX Runtime 1.14 - No AMX . 105.53 |========================================== ONNX Runtime 1.14 use_dnnl . 104.58 |========================================= ONNX Runtime 1.14 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better ONNX Runtime 1.14 .......... MIN: 1900 AVG: 2904 MAX: 3509 ONNX Runtime 1.14 - No AMX . MIN: 1900 AVG: 2917 MAX: 3506 ONNX Runtime 1.14 use_dnnl . MIN: 1900 AVG: 2921 MAX: 3515 ONNX Runtime 1.14 CPU Power Consumption Monitor Watts < Lower Is Better ONNX Runtime 1.14 .......... MIN: 198 AVG: 611 MAX: 637 ONNX Runtime 1.14 - No AMX . MIN: 210 AVG: 612 MAX: 639 ONNX Runtime 1.14 use_dnnl . MIN: 203 AVG: 612 MAX: 638 ONNX Runtime 1.14 CPU Temperature Monitor Celsius < Lower Is Better ONNX Runtime 1.14 .......... MIN: 36.0 AVG: 47.2 MAX: 50.0 ONNX Runtime 1.14 - No AMX . MIN: 37.0 AVG: 47.7 MAX: 50.0 ONNX Runtime 1.14 use_dnnl . MIN: 36.0 AVG: 47.5 MAX: 50.0 ONNX Runtime 1.14 Model: fcn-resnet101-11 - Device: CPU - Executor: Standard Inferences Per Second > Higher Is Better ONNX Runtime 1.14 .......... 10.00997 |====================================== ONNX Runtime 1.14 - No AMX . 10.46180 |======================================== ONNX Runtime 1.14 use_dnnl . 9.93450 |====================================== ONNX Runtime 1.14 Model: fcn-resnet101-11 - Device: CPU - Executor: Standard Inference Time Cost (ms) < Lower Is Better ONNX Runtime 1.14 .......... 100.29 |========================================== ONNX Runtime 1.14 - No AMX . 95.60 |======================================== ONNX Runtime 1.14 use_dnnl . 101.02 |========================================== ONNX Runtime 1.14 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better ONNX Runtime 1.14 .......... MIN: 1900 AVG: 2753 MAX: 3514 ONNX Runtime 1.14 - No AMX . MIN: 1900 AVG: 2751 MAX: 3515 ONNX Runtime 1.14 use_dnnl . MIN: 1900 AVG: 2771 MAX: 3511 ONNX Runtime 1.14 CPU Power Consumption Monitor Watts < Lower Is Better ONNX Runtime 1.14 .......... MIN: 122 AVG: 669 MAX: 709 ONNX Runtime 1.14 - No AMX . MIN: 203 AVG: 674 MAX: 707 ONNX Runtime 1.14 use_dnnl . MIN: 195 AVG: 669 MAX: 709 ONNX Runtime 1.14 CPU Temperature Monitor Celsius < Lower Is Better ONNX Runtime 1.14 .......... MIN: 36.0 AVG: 51.5 MAX: 54.0 ONNX Runtime 1.14 - No AMX . MIN: 36.0 AVG: 51.0 MAX: 54.0 ONNX Runtime 1.14 use_dnnl . MIN: 36.0 AVG: 51.5 MAX: 54.0 ONNX Runtime 1.14 Model: fcn-resnet101-11 - Device: CPU - Executor: Parallel Inferences Per Second > Higher Is Better ONNX Runtime 1.14 .......... 3.03253 |========================================= ONNX Runtime 1.14 - No AMX . 2.95924 |======================================== ONNX Runtime 1.14 use_dnnl . 3.00654 |========================================= ONNX Runtime 1.14 Model: fcn-resnet101-11 - Device: CPU - Executor: Parallel Inference Time Cost (ms) < Lower Is Better ONNX Runtime 1.14 .......... 329.87 |========================================= ONNX Runtime 1.14 - No AMX . 337.92 |========================================== ONNX Runtime 1.14 use_dnnl . 333.10 |========================================= ONNX Runtime 1.14 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better ONNX Runtime 1.14 .......... MIN: 1900 AVG: 2922 MAX: 3551 ONNX Runtime 1.14 - No AMX . MIN: 1900 AVG: 2905 MAX: 3508 ONNX Runtime 1.14 use_dnnl . MIN: 1900 AVG: 2916 MAX: 3677 ONNX Runtime 1.14 CPU Power Consumption Monitor Watts < Lower Is Better ONNX Runtime 1.14 .......... MIN: 195 AVG: 611 MAX: 638 ONNX Runtime 1.14 - No AMX . MIN: 210 AVG: 611 MAX: 638 ONNX Runtime 1.14 use_dnnl . MIN: 199 AVG: 613 MAX: 641 ONNX Runtime 1.14 CPU Temperature Monitor Celsius < Lower Is Better ONNX Runtime 1.14 .......... MIN: 39.0 AVG: 48.2 MAX: 50.0 ONNX Runtime 1.14 - No AMX . MIN: 39.0 AVG: 48.5 MAX: 50.0 ONNX Runtime 1.14 use_dnnl . MIN: 38.0 AVG: 48.5 MAX: 51.0 ONNX Runtime 1.14 Model: super-resolution-10 - Device: CPU - Executor: Standard Inferences Per Second > Higher Is Better ONNX Runtime 1.14 .......... 209.66 |========================================== ONNX Runtime 1.14 - No AMX . 209.70 |========================================== ONNX Runtime 1.14 use_dnnl . 211.42 |========================================== ONNX Runtime 1.14 Model: super-resolution-10 - Device: CPU - Executor: Standard Inference Time Cost (ms) < Lower Is Better ONNX Runtime 1.14 .......... 4.76912 |========================================= ONNX Runtime 1.14 - No AMX . 4.76804 |========================================= ONNX Runtime 1.14 use_dnnl . 4.72937 |========================================= ONNX Runtime 1.14 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better ONNX Runtime 1.14 .......... MIN: 1900 AVG: 3457 MAX: 3530 ONNX Runtime 1.14 - No AMX . MIN: 1900 AVG: 3450 MAX: 3516 ONNX Runtime 1.14 use_dnnl . MIN: 1900 AVG: 3457 MAX: 3517 ONNX Runtime 1.14 CPU Power Consumption Monitor Watts < Lower Is Better ONNX Runtime 1.14 .......... MIN: 209 AVG: 437 MAX: 451 ONNX Runtime 1.14 - No AMX . MIN: 199 AVG: 439 MAX: 454 ONNX Runtime 1.14 use_dnnl . MIN: 199 AVG: 440 MAX: 455 ONNX Runtime 1.14 CPU Temperature Monitor Celsius < Lower Is Better ONNX Runtime 1.14 .......... MIN: 37.0 AVG: 53.0 MAX: 55.0 ONNX Runtime 1.14 - No AMX . MIN: 37.0 AVG: 53.2 MAX: 55.0 ONNX Runtime 1.14 use_dnnl . MIN: 36.0 AVG: 53.2 MAX: 55.0 ONNX Runtime 1.14 Model: super-resolution-10 - Device: CPU - Executor: Parallel Inferences Per Second > Higher Is Better ONNX Runtime 1.14 .......... 179.90 |========================================== ONNX Runtime 1.14 - No AMX . 180.39 |========================================== ONNX Runtime 1.14 use_dnnl . 178.53 |========================================== ONNX Runtime 1.14 Model: super-resolution-10 - Device: CPU - Executor: Parallel Inference Time Cost (ms) < Lower Is Better ONNX Runtime 1.14 .......... 5.55742 |========================================= ONNX Runtime 1.14 - No AMX . 5.54240 |========================================= ONNX Runtime 1.14 use_dnnl . 5.60021 |========================================= ONNX Runtime 1.14 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better ONNX Runtime 1.14 .......... MIN: 1900 AVG: 3495 MAX: 3515 ONNX Runtime 1.14 - No AMX . MIN: 1900 AVG: 3478 MAX: 3517 ONNX Runtime 1.14 use_dnnl . MIN: 1900 AVG: 3489 MAX: 3514 ONNX Runtime 1.14 CPU Power Consumption Monitor Watts < Lower Is Better ONNX Runtime 1.14 .......... MIN: 192 AVG: 447 MAX: 462 ONNX Runtime 1.14 - No AMX . MIN: 197 AVG: 446 MAX: 460 ONNX Runtime 1.14 use_dnnl . MIN: 205 AVG: 446 MAX: 461 ONNX Runtime 1.14 CPU Temperature Monitor Celsius < Lower Is Better ONNX Runtime 1.14 .......... MIN: 34.0 AVG: 37.5 MAX: 40.0 ONNX Runtime 1.14 - No AMX . MIN: 35.0 AVG: 40.9 MAX: 45.0 ONNX Runtime 1.14 use_dnnl . MIN: 35.0 AVG: 39.1 MAX: 45.0 ONNX Runtime 1.14 Model: bertsquad-12 - Device: CPU - Executor: Standard Inferences Per Second > Higher Is Better ONNX Runtime 1.14 .......... 14.94 |========================================== ONNX Runtime 1.14 - No AMX . 15.17 |=========================================== ONNX Runtime 1.14 use_dnnl . 15.19 |=========================================== ONNX Runtime 1.14 Model: bertsquad-12 - Device: CPU - Executor: Standard Inference Time Cost (ms) < Lower Is Better ONNX Runtime 1.14 .......... 66.97 |=========================================== ONNX Runtime 1.14 - No AMX . 65.92 |========================================== ONNX Runtime 1.14 use_dnnl . 65.88 |========================================== ONNX Runtime 1.14 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better ONNX Runtime 1.14 .......... MIN: 1900 AVG: 3460 MAX: 3512 ONNX Runtime 1.14 - No AMX . MIN: 1900 AVG: 3470 MAX: 3512 ONNX Runtime 1.14 use_dnnl . MIN: 1900 AVG: 3466 MAX: 3516 ONNX Runtime 1.14 CPU Power Consumption Monitor Watts < Lower Is Better ONNX Runtime 1.14 .......... MIN: 201 AVG: 562 MAX: 584 ONNX Runtime 1.14 - No AMX . MIN: 196 AVG: 564 MAX: 586 ONNX Runtime 1.14 use_dnnl . MIN: 199 AVG: 566 MAX: 588 ONNX Runtime 1.14 CPU Temperature Monitor Celsius < Lower Is Better ONNX Runtime 1.14 .......... MIN: 32.0 AVG: 47.6 MAX: 50.0 ONNX Runtime 1.14 - No AMX . MIN: 33.0 AVG: 47.9 MAX: 51.0 ONNX Runtime 1.14 use_dnnl . MIN: 32.0 AVG: 49.2 MAX: 51.0 ONNX Runtime 1.14 Model: bertsquad-12 - Device: CPU - Executor: Parallel Inferences Per Second > Higher Is Better ONNX Runtime 1.14 .......... 15.03 |=========================================== ONNX Runtime 1.14 - No AMX . 15.00 |=========================================== ONNX Runtime 1.14 use_dnnl . 15.00 |=========================================== ONNX Runtime 1.14 Model: bertsquad-12 - Device: CPU - Executor: Parallel Inference Time Cost (ms) < Lower Is Better ONNX Runtime 1.14 .......... 66.55 |=========================================== ONNX Runtime 1.14 - No AMX . 66.69 |=========================================== ONNX Runtime 1.14 use_dnnl . 66.69 |=========================================== ONNX Runtime 1.14 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better ONNX Runtime 1.14 .......... MIN: 1900 AVG: 3179 MAX: 3512 ONNX Runtime 1.14 - No AMX . MIN: 1900 AVG: 3181 MAX: 3510 ONNX Runtime 1.14 use_dnnl . MIN: 1900 AVG: 3179 MAX: 3515 ONNX Runtime 1.14 CPU Power Consumption Monitor Watts < Lower Is Better ONNX Runtime 1.14 .......... MIN: 203 AVG: 562 MAX: 591 ONNX Runtime 1.14 - No AMX . MIN: 195 AVG: 560 MAX: 591 ONNX Runtime 1.14 use_dnnl . MIN: 213 AVG: 563 MAX: 593 ONNX Runtime 1.14 CPU Temperature Monitor Celsius < Lower Is Better ONNX Runtime 1.14 .......... MIN: 36.0 AVG: 45.3 MAX: 47.0 ONNX Runtime 1.14 - No AMX . MIN: 36.0 AVG: 45.1 MAX: 48.0 ONNX Runtime 1.14 use_dnnl . MIN: 37.0 AVG: 45.8 MAX: 49.0 ONNX Runtime 1.14 Model: GPT-2 - Device: CPU - Executor: Standard Inferences Per Second > Higher Is Better ONNX Runtime 1.14 .......... 190.02 |========================================= ONNX Runtime 1.14 - No AMX . 195.15 |========================================== ONNX Runtime 1.14 use_dnnl . 194.52 |========================================== ONNX Runtime 1.14 Model: GPT-2 - Device: CPU - Executor: Standard Inference Time Cost (ms) < Lower Is Better ONNX Runtime 1.14 .......... 5.26053 |========================================= ONNX Runtime 1.14 - No AMX . 5.13582 |======================================== ONNX Runtime 1.14 use_dnnl . 5.15414 |======================================== ONNX Runtime 1.14 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better ONNX Runtime 1.14 .......... MIN: 1900 AVG: 3482 MAX: 3515 ONNX Runtime 1.14 - No AMX . MIN: 1900 AVG: 3486 MAX: 3524 ONNX Runtime 1.14 use_dnnl . MIN: 1900 AVG: 3494 MAX: 3520 ONNX Runtime 1.14 CPU Power Consumption Monitor Watts < Lower Is Better ONNX Runtime 1.14 .......... MIN: 194 AVG: 435 MAX: 450 ONNX Runtime 1.14 - No AMX . MIN: 193 AVG: 436 MAX: 452 ONNX Runtime 1.14 use_dnnl . MIN: 105 AVG: 436 MAX: 453 ONNX Runtime 1.14 CPU Temperature Monitor Celsius < Lower Is Better ONNX Runtime 1.14 .......... MIN: 35.0 AVG: 48.1 MAX: 50.0 ONNX Runtime 1.14 - No AMX . MIN: 36.0 AVG: 48.9 MAX: 51.0 ONNX Runtime 1.14 use_dnnl . MIN: 35.0 AVG: 49.3 MAX: 52.0 ONNX Runtime 1.14 Model: GPT-2 - Device: CPU - Executor: Parallel Inferences Per Second > Higher Is Better ONNX Runtime 1.14 .......... 153.60 |========================================= ONNX Runtime 1.14 - No AMX . 158.31 |========================================== ONNX Runtime 1.14 use_dnnl . 150.86 |======================================== ONNX Runtime 1.14 Model: GPT-2 - Device: CPU - Executor: Parallel Inference Time Cost (ms) < Lower Is Better ONNX Runtime 1.14 .......... 6.50527 |======================================== ONNX Runtime 1.14 - No AMX . 6.31109 |======================================= ONNX Runtime 1.14 use_dnnl . 6.62309 |========================================= ONNX Runtime 1.14 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better ONNX Runtime 1.14 .......... MIN: 1900 AVG: 3494 MAX: 3516 ONNX Runtime 1.14 - No AMX . MIN: 1900 AVG: 3475 MAX: 3510 ONNX Runtime 1.14 use_dnnl . MIN: 1900 AVG: 3478 MAX: 3511 ONNX Runtime 1.14 CPU Power Consumption Monitor Watts < Lower Is Better ONNX Runtime 1.14 .......... MIN: 202 AVG: 439 MAX: 456 ONNX Runtime 1.14 - No AMX . MIN: 201 AVG: 439 MAX: 455 ONNX Runtime 1.14 use_dnnl . MIN: 204 AVG: 439 MAX: 459 ONNX Runtime 1.14 CPU Temperature Monitor Celsius < Lower Is Better ONNX Runtime 1.14 .......... MIN: 34.0 AVG: 37.4 MAX: 42.0 ONNX Runtime 1.14 - No AMX . MIN: 34.0 AVG: 37.5 MAX: 40.0 ONNX Runtime 1.14 use_dnnl . MIN: 34.0 AVG: 38.1 MAX: 42.0 ONNX Runtime 1.14 Model: ArcFace ResNet-100 - Device: CPU - Executor: Standard Inferences Per Second > Higher Is Better ONNX Runtime 1.14 .......... 33.97 |=========================================== ONNX Runtime 1.14 use_dnnl . 32.61 |========================================= ONNX Runtime 1.14 Model: ArcFace ResNet-100 - Device: CPU - Executor: Standard Inference Time Cost (ms) < Lower Is Better ONNX Runtime 1.14 .......... 29.44 |========================================= ONNX Runtime 1.14 use_dnnl . 30.68 |=========================================== ONNX Runtime 1.14 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better ONNX Runtime 1.14 .......... MIN: 2900 AVG: 2935 MAX: 3514 ONNX Runtime 1.14 use_dnnl . MIN: 1900 AVG: 2932 MAX: 3512 ONNX Runtime 1.14 CPU Power Consumption Monitor Watts < Lower Is Better ONNX Runtime 1.14 .......... MIN: 202 AVG: 622 MAX: 649 ONNX Runtime 1.14 use_dnnl . MIN: 199 AVG: 621 MAX: 652 ONNX Runtime 1.14 CPU Temperature Monitor Celsius < Lower Is Better ONNX Runtime 1.14 .......... MIN: 33.0 AVG: 48.4 MAX: 52.0 ONNX Runtime 1.14 use_dnnl . MIN: 32.0 AVG: 48.9 MAX: 52.0 ONNX Runtime 1.14 Model: ArcFace ResNet-100 - Device: CPU - Executor: Parallel Inferences Per Second > Higher Is Better ONNX Runtime 1.14 .......... 27.21 |=========================================== ONNX Runtime 1.14 use_dnnl . 27.43 |=========================================== ONNX Runtime 1.14 Model: ArcFace ResNet-100 - Device: CPU - Executor: Parallel Inference Time Cost (ms) < Lower Is Better ONNX Runtime 1.14 .......... 36.74 |=========================================== ONNX Runtime 1.14 use_dnnl . 36.47 |=========================================== ONNX Runtime 1.14 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better ONNX Runtime 1.14 .......... MIN: 1900 AVG: 2911 MAX: 3514 ONNX Runtime 1.14 use_dnnl . MIN: 1900 AVG: 2929 MAX: 3510 ONNX Runtime 1.14 CPU Power Consumption Monitor Watts < Lower Is Better ONNX Runtime 1.14 .......... MIN: 207 AVG: 623 MAX: 650 ONNX Runtime 1.14 use_dnnl . MIN: 201 AVG: 625 MAX: 651 ONNX Runtime 1.14 CPU Temperature Monitor Celsius < Lower Is Better ONNX Runtime 1.14 .......... MIN: 37.0 AVG: 47.9 MAX: 50.0 ONNX Runtime 1.14 use_dnnl . MIN: 37.0 AVG: 48.4 MAX: 51.0 ONNX Runtime 1.14 Model: ResNet50 v1-12-int8 - Device: CPU - Executor: Standard Inferences Per Second > Higher Is Better ONNX Runtime 1.14 .......... 177.93 |========================================== ONNX Runtime 1.14 use_dnnl . 175.67 |========================================= ONNX Runtime 1.14 Model: ResNet50 v1-12-int8 - Device: CPU - Executor: Standard Inference Time Cost (ms) < Lower Is Better ONNX Runtime 1.14 .......... 5.61969 |======================================== ONNX Runtime 1.14 use_dnnl . 5.69175 |========================================= ONNX Runtime 1.14 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better ONNX Runtime 1.14 .......... MIN: 1900 AVG: 2904 MAX: 3506 ONNX Runtime 1.14 use_dnnl . MIN: 1900 AVG: 2905 MAX: 3503 ONNX Runtime 1.14 CPU Power Consumption Monitor Watts < Lower Is Better ONNX Runtime 1.14 .......... MIN: 202 AVG: 616 MAX: 637 ONNX Runtime 1.14 use_dnnl . MIN: 208 AVG: 617 MAX: 639 ONNX Runtime 1.14 CPU Temperature Monitor Celsius < Lower Is Better ONNX Runtime 1.14 .......... MIN: 36.0 AVG: 50.1 MAX: 52.0 ONNX Runtime 1.14 use_dnnl . MIN: 36.0 AVG: 50.7 MAX: 53.0 ONNX Runtime 1.14 Model: ResNet50 v1-12-int8 - Device: CPU - Executor: Parallel Inferences Per Second > Higher Is Better ONNX Runtime 1.14 .......... 162.48 |========================================= ONNX Runtime 1.14 use_dnnl . 164.90 |========================================== ONNX Runtime 1.14 Model: ResNet50 v1-12-int8 - Device: CPU - Executor: Parallel Inference Time Cost (ms) < Lower Is Better ONNX Runtime 1.14 .......... 6.15564 |========================================= ONNX Runtime 1.14 use_dnnl . 6.06317 |======================================== ONNX Runtime 1.14 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better ONNX Runtime 1.14 .......... MIN: 1900 AVG: 2890 MAX: 3666 ONNX Runtime 1.14 use_dnnl . MIN: 1900 AVG: 2889 MAX: 3507 ONNX Runtime 1.14 CPU Power Consumption Monitor Watts < Lower Is Better ONNX Runtime 1.14 .......... MIN: 290 AVG: 633 MAX: 659 ONNX Runtime 1.14 use_dnnl . MIN: 289 AVG: 638 MAX: 660 ONNX Runtime 1.14 CPU Temperature Monitor Celsius < Lower Is Better ONNX Runtime 1.14 .......... MIN: 38.0 AVG: 50.7 MAX: 53.0 ONNX Runtime 1.14 use_dnnl . MIN: 40.0 AVG: 48.9 MAX: 51.0 ONNX Runtime 1.14 Model: CaffeNet 12-int8 - Device: CPU - Executor: Standard Inferences Per Second > Higher Is Better ONNX Runtime 1.14 .......... 684.15 |========================================= ONNX Runtime 1.14 use_dnnl . 695.78 |========================================== ONNX Runtime 1.14 Model: CaffeNet 12-int8 - Device: CPU - Executor: Standard Inference Time Cost (ms) < Lower Is Better ONNX Runtime 1.14 .......... 1.46149 |========================================= ONNX Runtime 1.14 use_dnnl . 1.43703 |======================================== ONNX Runtime 1.14 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better ONNX Runtime 1.14 .......... MIN: 1900 AVG: 2908 MAX: 3400 ONNX Runtime 1.14 use_dnnl . MIN: 1900 AVG: 2914 MAX: 3501 ONNX Runtime 1.14 CPU Power Consumption Monitor Watts < Lower Is Better ONNX Runtime 1.14 .......... MIN: 289 AVG: 602 MAX: 623 ONNX Runtime 1.14 use_dnnl . MIN: 283 AVG: 607 MAX: 628 ONNX Runtime 1.14 CPU Temperature Monitor Celsius < Lower Is Better ONNX Runtime 1.14 .......... MIN: 40.0 AVG: 48.8 MAX: 51.0 ONNX Runtime 1.14 use_dnnl . MIN: 40.0 AVG: 49.7 MAX: 51.0 ONNX Runtime 1.14 Model: CaffeNet 12-int8 - Device: CPU - Executor: Parallel Inferences Per Second > Higher Is Better ONNX Runtime 1.14 .......... 589.20 |========================================== ONNX Runtime 1.14 use_dnnl . 590.80 |========================================== ONNX Runtime 1.14 Model: CaffeNet 12-int8 - Device: CPU - Executor: Parallel Inference Time Cost (ms) < Lower Is Better ONNX Runtime 1.14 .......... 1.69517 |========================================= ONNX Runtime 1.14 use_dnnl . 1.69067 |========================================= ONNX Runtime 1.14 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better ONNX Runtime 1.14 .......... MIN: 1900 AVG: 2908 MAX: 3500 ONNX Runtime 1.14 use_dnnl . MIN: 1900 AVG: 2903 MAX: 3501 ONNX Runtime 1.14 CPU Power Consumption Monitor Watts < Lower Is Better ONNX Runtime 1.14 .......... MIN: 278 AVG: 615 MAX: 633 ONNX Runtime 1.14 use_dnnl . MIN: 272 AVG: 617 MAX: 637 ONNX Runtime 1.14 CPU Temperature Monitor Celsius < Lower Is Better ONNX Runtime 1.14 .......... MIN: 40.0 AVG: 46.9 MAX: 49.0 ONNX Runtime 1.14 use_dnnl . MIN: 40.0 AVG: 48.1 MAX: 51.0 ONNX Runtime 1.14 Model: Faster R-CNN R-50-FPN-int8 - Device: CPU - Executor: Standard Inferences Per Second > Higher Is Better ONNX Runtime 1.14 .......... 39.77 |=========================================== ONNX Runtime 1.14 use_dnnl . 40.01 |=========================================== ONNX Runtime 1.14 Model: Faster R-CNN R-50-FPN-int8 - Device: CPU - Executor: Standard Inference Time Cost (ms) < Lower Is Better ONNX Runtime 1.14 .......... 25.14 |=========================================== ONNX Runtime 1.14 use_dnnl . 25.00 |=========================================== ONNX Runtime 1.14 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better ONNX Runtime 1.14 .......... MIN: 1900 AVG: 2982 MAX: 3510 ONNX Runtime 1.14 use_dnnl . MIN: 1900 AVG: 2997 MAX: 3509 ONNX Runtime 1.14 CPU Power Consumption Monitor Watts < Lower Is Better ONNX Runtime 1.14 .......... MIN: 240 AVG: 582 MAX: 599 ONNX Runtime 1.14 use_dnnl . MIN: 258 AVG: 583 MAX: 603 ONNX Runtime 1.14 CPU Temperature Monitor Celsius < Lower Is Better ONNX Runtime 1.14 .......... MIN: 37.0 AVG: 46.3 MAX: 48.0 ONNX Runtime 1.14 use_dnnl . MIN: 40.0 AVG: 47.9 MAX: 50.0 ONNX Runtime 1.14 Model: Faster R-CNN R-50-FPN-int8 - Device: CPU - Executor: Parallel Inferences Per Second > Higher Is Better ONNX Runtime 1.14 .......... 32.90 |=========================================== ONNX Runtime 1.14 use_dnnl . 31.78 |========================================== ONNX Runtime 1.14 Model: Faster R-CNN R-50-FPN-int8 - Device: CPU - Executor: Parallel Inference Time Cost (ms) < Lower Is Better ONNX Runtime 1.14 .......... 30.40 |========================================== ONNX Runtime 1.14 use_dnnl . 31.47 |=========================================== ONNX Runtime 1.14 CPU Peak Freq (Highest CPU Core Frequency) Monitor Megahertz > Higher Is Better ONNX Runtime 1.14 .......... MIN: 1900 AVG: 3070 MAX: 3525 ONNX Runtime 1.14 use_dnnl . MIN: 1900 AVG: 3034 MAX: 3508 ONNX Runtime 1.14 CPU Power Consumption Monitor Watts < Lower Is Better ONNX Runtime 1.14 .......... MIN: 237 AVG: 562 MAX: 589 ONNX Runtime 1.14 use_dnnl . MIN: 268 AVG: 562 MAX: 588 ONNX Runtime 1.14 CPU Temperature Monitor Celsius < Lower Is Better ONNX Runtime 1.14 .......... MIN: 37.0 AVG: 45.1 MAX: 48.0 ONNX Runtime 1.14 use_dnnl . MIN: 40.0 AVG: 46.0 MAX: 51.0