ds AMD Ryzen 9 7950X 16-Core testing with a ASUS ROG STRIX X670E-E GAMING WIFI (1416 BIOS) and NVIDIA NV174 8GB on Ubuntu 23.10 via the Phoronix Test Suite. a: Processor: AMD Ryzen 9 7950X 16-Core @ 5.88GHz (16 Cores / 32 Threads), Motherboard: ASUS ROG STRIX X670E-E GAMING WIFI (1416 BIOS), Chipset: AMD Device 14d8, Memory: 32GB, Disk: 2000GB Samsung SSD 980 PRO 2TB + 4001GB Western Digital WD_BLACK SN850X 4000GB, Graphics: NVIDIA NV174 8GB, Audio: NVIDIA GA104 HD Audio, Monitor: DELL U2723QE, Network: Intel I225-V + Intel Wi-Fi 6 AX210/AX211/AX411 OS: Ubuntu 23.10, Kernel: 6.7.0-060700rc2daily20231127-generic (x86_64), Desktop: GNOME Shell 45.1, Display Server: X Server 1.21.1.7 + Wayland, Display Driver: nouveau, OpenGL: 4.3 Mesa 24.0~git2311260600.945288~oibaf~m (git-945288f 2023-11-26 mantic-oibaf-ppa), Compiler: GCC 13.2.0 + LLVM 16.0.6, File-System: ext4, Screen Resolution: 3840x2160 b: Processor: AMD Ryzen 9 7950X 16-Core @ 5.88GHz (16 Cores / 32 Threads), Motherboard: ASUS ROG STRIX X670E-E GAMING WIFI (1416 BIOS), Chipset: AMD Device 14d8, Memory: 32GB, Disk: 2000GB Samsung SSD 980 PRO 2TB + 4001GB Western Digital WD_BLACK SN850X 4000GB, Graphics: NVIDIA NV174 8GB, Audio: NVIDIA GA104 HD Audio, Monitor: DELL U2723QE, Network: Intel I225-V + Intel Wi-Fi 6 AX210/AX211/AX411 OS: Ubuntu 23.10, Kernel: 6.7.0-060700rc2daily20231127-generic (x86_64), Desktop: GNOME Shell 45.1, Display Server: X Server 1.21.1.7 + Wayland, Display Driver: nouveau, OpenGL: 4.3 Mesa 24.0~git2311260600.945288~oibaf~m (git-945288f 2023-11-26 mantic-oibaf-ppa), Compiler: GCC 13.2.0 + LLVM 16.0.6, File-System: ext4, Screen Resolution: 3840x2160 c: Processor: AMD Ryzen 9 7950X 16-Core @ 5.88GHz (16 Cores / 32 Threads), Motherboard: ASUS ROG STRIX X670E-E GAMING WIFI (1416 BIOS), Chipset: AMD Device 14d8, Memory: 32GB, Disk: 2000GB Samsung SSD 980 PRO 2TB + 4001GB Western Digital WD_BLACK SN850X 4000GB, Graphics: NVIDIA NV174 8GB, Audio: NVIDIA GA104 HD Audio, Monitor: DELL U2723QE, Network: Intel I225-V + Intel Wi-Fi 6 AX210/AX211/AX411 OS: Ubuntu 23.10, Kernel: 6.7.0-060700rc2daily20231127-generic (x86_64), Desktop: GNOME Shell 45.1, Display Server: X Server 1.21.1.7 + Wayland, Display Driver: nouveau, OpenGL: 4.3 Mesa 24.0~git2311260600.945288~oibaf~m (git-945288f 2023-11-26 mantic-oibaf-ppa), Compiler: GCC 13.2.0 + LLVM 16.0.6, File-System: ext4, Screen Resolution: 3840x2160 d: Processor: AMD Ryzen 9 7950X 16-Core @ 5.88GHz (16 Cores / 32 Threads), Motherboard: ASUS ROG STRIX X670E-E GAMING WIFI (1416 BIOS), Chipset: AMD Device 14d8, Memory: 32GB, Disk: 2000GB Samsung SSD 980 PRO 2TB + 4001GB Western Digital WD_BLACK SN850X 4000GB, Graphics: NVIDIA NV174 8GB, Audio: NVIDIA GA104 HD Audio, Monitor: DELL U2723QE, Network: Intel I225-V + Intel Wi-Fi 6 AX210/AX211/AX411 OS: Ubuntu 23.10, Kernel: 6.7.0-060700rc2daily20231127-generic (x86_64), Desktop: GNOME Shell 45.1, Display Server: X Server 1.21.1.7 + Wayland, Display Driver: nouveau, OpenGL: 4.3 Mesa 24.0~git2311260600.945288~oibaf~m (git-945288f 2023-11-26 mantic-oibaf-ppa), Compiler: GCC 13.2.0 + LLVM 16.0.6, File-System: ext4, Screen Resolution: 3840x2160 Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 22.74 |==================================================================== b . 22.69 |==================================================================== c . 22.70 |==================================================================== d . 22.62 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 19.57 |==================================================================== b . 19.54 |==================================================================== c . 19.55 |==================================================================== d . 19.57 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 983.31 |=================================================================== b . 988.30 |=================================================================== c . 986.68 |=================================================================== d . 987.41 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 299.91 |=================================================================== b . 301.51 |=================================================================== c . 301.17 |=================================================================== d . 301.65 |=================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 296.04 |=================================================================== b . 295.52 |=================================================================== c . 295.73 |=================================================================== d . 295.34 |=================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 197.08 |=================================================================== b . 198.48 |=================================================================== c . 198.25 |=================================================================== d . 198.36 |=================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 2314.23 |================================================================== b . 2315.52 |================================================================== c . 2317.26 |================================================================== d . 2293.25 |================================================================= Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 1295.81 |================================================================= b . 1295.23 |================================================================= c . 1308.77 |================================================================== d . 1299.09 |================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 127.71 |=================================================================== b . 127.25 |=================================================================== c . 127.47 |=================================================================== d . 128.02 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 101.87 |=================================================================== b . 101.77 |=================================================================== c . 101.88 |=================================================================== d . 101.48 |=================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 29.35 |==================================================================== b . 29.39 |==================================================================== c . 29.40 |==================================================================== d . 29.36 |==================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 21.13 |==================================================================== b . 21.14 |==================================================================== c . 21.13 |==================================================================== d . 21.24 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 296.03 |=================================================================== b . 295.76 |=================================================================== c . 295.24 |=================================================================== d . 295.57 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 198.90 |=================================================================== b . 198.45 |=================================================================== c . 198.77 |=================================================================== d . 198.97 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 133.54 |=================================================================== b . 133.17 |=================================================================== c . 132.03 |================================================================== d . 132.59 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 103.17 |=================================================================== b . 103.16 |=================================================================== c . 103.32 |=================================================================== d . 103.23 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 203.97 |=================================================================== b . 203.49 |=================================================================== c . 203.92 |=================================================================== d . 203.62 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 116.37 |================================================================== b . 116.58 |=================================================================== c . 116.71 |=================================================================== d . 117.26 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 38.98 |==================================================================== b . 38.87 |==================================================================== c . 39.04 |==================================================================== d . 39.13 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 32.39 |==================================================================== b . 32.42 |==================================================================== c . 32.40 |==================================================================== d . 32.41 |==================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 462.42 |=================================================================== b . 462.68 |=================================================================== c . 461.53 |=================================================================== d . 462.41 |=================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 110.62 |=================================================================== b . 110.03 |=================================================================== c . 110.06 |=================================================================== d . 110.50 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 22.76 |==================================================================== b . 22.72 |==================================================================== c . 22.69 |==================================================================== d . 22.71 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 19.54 |==================================================================== b . 19.53 |==================================================================== c . 19.53 |==================================================================== d . 19.51 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 350.75 |=================================================================== b . 351.56 |=================================================================== c . 351.40 |=================================================================== d . 352.48 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 51.10 |==================================================================== b . 51.17 |==================================================================== c . 51.16 |==================================================================== d . 51.10 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 8.1243 |=================================================================== b . 8.0835 |=================================================================== c . 8.0963 |=================================================================== d . 8.0908 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 3.3312 |=================================================================== b . 3.3138 |=================================================================== c . 3.3173 |=================================================================== d . 3.3119 |=================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 27.01 |==================================================================== b . 27.06 |==================================================================== c . 27.04 |==================================================================== d . 27.07 |==================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 5.0670 |=================================================================== b . 5.0320 |=================================================================== c . 5.0382 |=================================================================== d . 5.0356 |=================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 3.4471 |================================================================== b . 3.4442 |================================================================== c . 3.4420 |================================================================== d . 3.4780 |=================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 0.7695 |=================================================================== b . 0.7698 |=================================================================== c . 0.7620 |================================================================== d . 0.7672 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 62.63 |==================================================================== b . 62.86 |==================================================================== c . 62.73 |==================================================================== d . 62.47 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 9.8076 |=================================================================== b . 9.8180 |=================================================================== c . 9.8068 |=================================================================== d . 9.8453 |=================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 272.17 |=================================================================== b . 271.91 |=================================================================== c . 271.78 |=================================================================== d . 272.15 |=================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 47.32 |==================================================================== b . 47.30 |==================================================================== c . 47.32 |==================================================================== d . 47.07 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 27.01 |==================================================================== b . 27.03 |==================================================================== c . 27.08 |==================================================================== d . 27.05 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 5.0217 |=================================================================== b . 5.0331 |=================================================================== c . 5.0250 |=================================================================== d . 5.0198 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 59.86 |=================================================================== b . 60.03 |=================================================================== c . 60.57 |==================================================================== d . 60.29 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 9.6870 |=================================================================== b . 9.6883 |=================================================================== c . 9.6736 |=================================================================== d . 9.6809 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 39.21 |==================================================================== b . 39.29 |==================================================================== c . 39.22 |==================================================================== d . 39.28 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 8.5884 |=================================================================== b . 8.5729 |=================================================================== c . 8.5631 |=================================================================== d . 8.5234 |================================================================== Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 204.86 |=================================================================== b . 205.60 |=================================================================== c . 204.58 |=================================================================== d . 204.22 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 30.86 |==================================================================== b . 30.83 |==================================================================== c . 30.85 |==================================================================== d . 30.85 |==================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 17.29 |==================================================================== b . 17.28 |==================================================================== c . 17.32 |==================================================================== d . 17.29 |==================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 9.0307 |=================================================================== b . 9.0802 |=================================================================== c . 9.0770 |=================================================================== d . 9.0414 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 349.98 |=================================================================== b . 351.51 |=================================================================== c . 351.44 |=================================================================== d . 351.41 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 51.17 |==================================================================== b . 51.20 |==================================================================== c . 51.19 |==================================================================== d . 51.25 |====================================================================