onnx tr AMD Ryzen Threadripper PRO 5965WX 24-Cores testing with a ASUS Pro WS WRX80E-SAGE SE WIFI (1201 BIOS) and ASUS NVIDIA NV106 2GB on Ubuntu 23.10 via the Phoronix Test Suite. a: Processor: AMD Ryzen Threadripper PRO 5965WX 24-Cores @ 3.80GHz (24 Cores / 48 Threads), Motherboard: ASUS Pro WS WRX80E-SAGE SE WIFI (1201 BIOS), Chipset: AMD Starship/Matisse, Memory: 8 x 16GB DDR4-2133MT/s Corsair CMK32GX4M2E3200C16, Disk: 2048GB SOLIDIGM SSDPFKKW020X7, Graphics: ASUS NVIDIA NV106 2GB, Audio: AMD Starship/Matisse, Monitor: VA2431, Network: 2 x Intel X550 + Intel Wi-Fi 6 AX200 OS: Ubuntu 23.10, Kernel: 6.5.0-13-generic (x86_64), Desktop: GNOME Shell 45.0, Display Server: X Server + Wayland, Display Driver: nouveau, OpenGL: 4.3 Mesa 23.2.1-1ubuntu3, Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080 AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS: Processor: AMD Ryzen Threadripper PRO 5965WX 24-Cores @ 3.80GHz (24 Cores / 48 Threads), Motherboard: ASUS Pro WS WRX80E-SAGE SE WIFI (1201 BIOS), Chipset: AMD Starship/Matisse, Memory: 8 x 16GB DDR4-2133MT/s Corsair CMK32GX4M2E3200C16, Disk: 2048GB SOLIDIGM SSDPFKKW020X7, Graphics: ASUS NVIDIA NV106 2GB, Audio: AMD Starship/Matisse, Monitor: VA2431, Network: 2 x Intel X550 + Intel Wi-Fi 6 AX200 OS: Ubuntu 23.10, Kernel: 6.5.0-13-generic (x86_64), Desktop: GNOME Shell 45.0, Display Server: X Server + Wayland, Display Driver: nouveau, OpenGL: 4.3 Mesa 23.2.1-1ubuntu3, Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080 c: Processor: AMD Ryzen Threadripper PRO 5965WX 24-Cores @ 3.80GHz (24 Cores / 48 Threads), Motherboard: ASUS Pro WS WRX80E-SAGE SE WIFI (1201 BIOS), Chipset: AMD Starship/Matisse, Memory: 8 x 16GB DDR4-2133MT/s Corsair CMK32GX4M2E3200C16, Disk: 2048GB SOLIDIGM SSDPFKKW020X7, Graphics: ASUS NVIDIA NV106 2GB, Audio: AMD Starship/Matisse, Monitor: VA2431, Network: 2 x Intel X550 + Intel Wi-Fi 6 AX200 OS: Ubuntu 23.10, Kernel: 6.5.0-13-generic (x86_64), Desktop: GNOME Shell 45.0, Display Server: X Server + Wayland, Display Driver: nouveau, OpenGL: 4.3 Mesa 23.2.1-1ubuntu3, Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080 ONNX Runtime 1.17 Model: GPT-2 - Device: CPU - Executor: Parallel Inferences Per Second > Higher Is Better a ................................................. 131.03 |================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 137.54 |=================== c ................................................. 131.26 |================== ONNX Runtime 1.17 Model: GPT-2 - Device: CPU - Executor: Parallel Inference Time Cost (ms) < Lower Is Better a ................................................. 7.62456 |================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 7.26342 |================= c ................................................. 7.61116 |================== ONNX Runtime 1.17 Model: GPT-2 - Device: CPU - Executor: Standard Inferences Per Second > Higher Is Better a ................................................. 137.64 |================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 145.11 |=================== c ................................................. 146.78 |=================== ONNX Runtime 1.17 Model: GPT-2 - Device: CPU - Executor: Standard Inference Time Cost (ms) < Lower Is Better a ................................................. 7.26082 |================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 6.88710 |================= c ................................................. 6.86398 |================= ONNX Runtime 1.17 Model: yolov4 - Device: CPU - Executor: Parallel Inferences Per Second > Higher Is Better a ................................................. 9.54753 |================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 9.15942 |================= c ................................................. 9.49134 |================== ONNX Runtime 1.17 Model: yolov4 - Device: CPU - Executor: Parallel Inference Time Cost (ms) < Lower Is Better a ................................................. 104.74 |================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 109.17 |=================== c ................................................. 105.36 |================== ONNX Runtime 1.17 Model: yolov4 - Device: CPU - Executor: Standard Inferences Per Second > Higher Is Better a ................................................. 10.34020 |================= AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 9.18180 |=============== c ................................................. 9.70516 |================ ONNX Runtime 1.17 Model: yolov4 - Device: CPU - Executor: Standard Inference Time Cost (ms) < Lower Is Better a ................................................. 96.71 |================= AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 108.91 |=================== c ................................................. 103.65 |================== ONNX Runtime 1.17 Model: T5 Encoder - Device: CPU - Executor: Parallel Inferences Per Second > Higher Is Better a ................................................. 181.53 |=================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 173.72 |================== c ................................................. 176.90 |=================== ONNX Runtime 1.17 Model: T5 Encoder - Device: CPU - Executor: Parallel Inference Time Cost (ms) < Lower Is Better a ................................................. 5.50730 |================= AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 5.75502 |================== c ................................................. 5.65170 |================== ONNX Runtime 1.17 Model: T5 Encoder - Device: CPU - Executor: Standard Inferences Per Second > Higher Is Better a ................................................. 201.08 |=================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 197.04 |=================== c ................................................. 193.94 |================== ONNX Runtime 1.17 Model: T5 Encoder - Device: CPU - Executor: Standard Inference Time Cost (ms) < Lower Is Better a ................................................. 4.97262 |================= AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 5.07413 |================== c ................................................. 5.15764 |================== ONNX Runtime 1.17 Model: bertsquad-12 - Device: CPU - Executor: Parallel Inferences Per Second > Higher Is Better a ................................................. 12.53 |==================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 12.73 |==================== c ................................................. 12.47 |==================== ONNX Runtime 1.17 Model: bertsquad-12 - Device: CPU - Executor: Parallel Inference Time Cost (ms) < Lower Is Better a ................................................. 79.80 |==================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 78.56 |==================== c ................................................. 80.21 |==================== ONNX Runtime 1.17 Model: bertsquad-12 - Device: CPU - Executor: Standard Inferences Per Second > Higher Is Better a ................................................. 13.09 |==================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 11.91 |================== c ................................................. 13.35 |==================== ONNX Runtime 1.17 Model: bertsquad-12 - Device: CPU - Executor: Standard Inference Time Cost (ms) < Lower Is Better a ................................................. 76.38 |================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 83.94 |==================== c ................................................. 75.73 |================== ONNX Runtime 1.17 Model: CaffeNet 12-int8 - Device: CPU - Executor: Parallel Inferences Per Second > Higher Is Better a ................................................. 687.28 |=================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 686.14 |=================== c ................................................. 692.18 |=================== ONNX Runtime 1.17 Model: CaffeNet 12-int8 - Device: CPU - Executor: Parallel Inference Time Cost (ms) < Lower Is Better a ................................................. 1.45330 |================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 1.45579 |================== c ................................................. 1.44343 |================== ONNX Runtime 1.17 Model: CaffeNet 12-int8 - Device: CPU - Executor: Standard Inferences Per Second > Higher Is Better a ................................................. 794.13 |=================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 662.19 |================ c ................................................. 763.04 |================== ONNX Runtime 1.17 Model: CaffeNet 12-int8 - Device: CPU - Executor: Standard Inference Time Cost (ms) < Lower Is Better a ................................................. 1.25873 |=============== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 1.50952 |================== c ................................................. 1.31876 |================ ONNX Runtime 1.17 Model: fcn-resnet101-11 - Device: CPU - Executor: Parallel Inferences Per Second > Higher Is Better a ................................................. 1.49191 |================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 1.44815 |================= c ................................................. 1.48414 |================== ONNX Runtime 1.17 Model: fcn-resnet101-11 - Device: CPU - Executor: Parallel Inference Time Cost (ms) < Lower Is Better a ................................................. 670.28 |================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 690.53 |=================== c ................................................. 673.79 |=================== ONNX Runtime 1.17 Model: fcn-resnet101-11 - Device: CPU - Executor: Standard Inferences Per Second > Higher Is Better a ................................................. 2.33114 |============== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 2.32498 |============== c ................................................. 2.95286 |================== ONNX Runtime 1.17 Model: fcn-resnet101-11 - Device: CPU - Executor: Standard Inference Time Cost (ms) < Lower Is Better a ................................................. 428.97 |=================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 430.11 |=================== c ................................................. 347.23 |=============== ONNX Runtime 1.17 Model: ArcFace ResNet-100 - Device: CPU - Executor: Parallel Inferences Per Second > Higher Is Better a ................................................. 29.95 |==================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 29.28 |=================== c ................................................. 30.41 |==================== ONNX Runtime 1.17 Model: ArcFace ResNet-100 - Device: CPU - Executor: Parallel Inference Time Cost (ms) < Lower Is Better a ................................................. 33.39 |==================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 34.15 |==================== c ................................................. 32.89 |=================== ONNX Runtime 1.17 Model: ArcFace ResNet-100 - Device: CPU - Executor: Standard Inferences Per Second > Higher Is Better a ................................................. 29.04 |================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 32.67 |==================== c ................................................. 32.28 |==================== ONNX Runtime 1.17 Model: ArcFace ResNet-100 - Device: CPU - Executor: Standard Inference Time Cost (ms) < Lower Is Better a ................................................. 34.43 |==================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 30.60 |================== c ................................................. 31.14 |================== ONNX Runtime 1.17 Model: ResNet50 v1-12-int8 - Device: CPU - Executor: Parallel Inferences Per Second > Higher Is Better a ................................................. 233.83 |=================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 233.76 |=================== c ................................................. 236.74 |=================== ONNX Runtime 1.17 Model: ResNet50 v1-12-int8 - Device: CPU - Executor: Parallel Inference Time Cost (ms) < Lower Is Better a ................................................. 4.27552 |================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 4.27661 |================== c ................................................. 4.22286 |================== ONNX Runtime 1.17 Model: ResNet50 v1-12-int8 - Device: CPU - Executor: Standard Inferences Per Second > Higher Is Better a ................................................. 263.28 |================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 254.40 |================== c ................................................. 275.56 |=================== ONNX Runtime 1.17 Model: ResNet50 v1-12-int8 - Device: CPU - Executor: Standard Inference Time Cost (ms) < Lower Is Better a ................................................. 3.79772 |================= AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 3.93035 |================== c ................................................. 3.64067 |================= ONNX Runtime 1.17 Model: super-resolution-10 - Device: CPU - Executor: Parallel Inferences Per Second > Higher Is Better a ................................................. 121.57 |=================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 119.33 |================== c ................................................. 123.18 |=================== ONNX Runtime 1.17 Model: super-resolution-10 - Device: CPU - Executor: Parallel Inference Time Cost (ms) < Lower Is Better a ................................................. 8.22445 |================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 8.37853 |================== c ................................................. 8.11762 |================= ONNX Runtime 1.17 Model: super-resolution-10 - Device: CPU - Executor: Standard Inferences Per Second > Higher Is Better a ................................................. 99.84 |==================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 95.02 |=================== ONNX Runtime 1.17 Model: super-resolution-10 - Device: CPU - Executor: Standard Inference Time Cost (ms) < Lower Is Better a ................................................. 10.02 |=================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 10.52 |==================== ONNX Runtime 1.17 Model: Faster R-CNN R-50-FPN-int8 - Device: CPU - Executor: Parallel Inferences Per Second > Higher Is Better a ................................................. 30.71 |==================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 31.14 |==================== ONNX Runtime 1.17 Model: Faster R-CNN R-50-FPN-int8 - Device: CPU - Executor: Parallel Inference Time Cost (ms) < Lower Is Better a ................................................. 32.57 |==================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 32.11 |==================== ONNX Runtime 1.17 Model: Faster R-CNN R-50-FPN-int8 - Device: CPU - Executor: Standard Inferences Per Second > Higher Is Better a ................................................. 35.09 |=================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 36.41 |==================== ONNX Runtime 1.17 Model: Faster R-CNN R-50-FPN-int8 - Device: CPU - Executor: Standard Inference Time Cost (ms) < Lower Is Better a ................................................. 28.49 |==================== AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS . 27.46 |===================