deepspaarse 17 AMD Ryzen 9 7950X 16-Core testing with a ASUS ROG STRIX X670E-E GAMING WIFI (1905 BIOS) and NVIDIA GeForce RTX 3080 10GB on Ubuntu 23.10 via the Phoronix Test Suite. a: Processor: AMD Ryzen 9 7950X 16-Core @ 5.88GHz (16 Cores / 32 Threads), Motherboard: ASUS ROG STRIX X670E-E GAMING WIFI (1905 BIOS), Chipset: AMD Device 14d8, Memory: 2 x 16GB DRAM-6000MT/s G Skill F5-6000J3038F16G, Disk: 2000GB Samsung SSD 980 PRO 2TB + Western Digital WD_BLACK SN850X 2000GB, Graphics: NVIDIA GeForce RTX 3080 10GB, Audio: NVIDIA GA102 HD Audio, Monitor: DELL U2723QE, Network: Intel I225-V + Intel Wi-Fi 6 AX210/AX211/AX411 OS: Ubuntu 23.10, Kernel: 6.7.0-060700-generic (x86_64), Desktop: GNOME Shell 45.2, Display Server: X Server 1.21.1.7, Display Driver: NVIDIA 550.54.14, OpenGL: 4.6.0, OpenCL: OpenCL 3.0 CUDA 12.4.89, Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 3840x2160 b: Processor: AMD Ryzen 9 7950X 16-Core @ 5.88GHz (16 Cores / 32 Threads), Motherboard: ASUS ROG STRIX X670E-E GAMING WIFI (1905 BIOS), Chipset: AMD Device 14d8, Memory: 2 x 16GB DRAM-6000MT/s G Skill F5-6000J3038F16G, Disk: 2000GB Samsung SSD 980 PRO 2TB + Western Digital WD_BLACK SN850X 2000GB, Graphics: NVIDIA GeForce RTX 3080 10GB, Audio: NVIDIA GA102 HD Audio, Monitor: DELL U2723QE, Network: Intel I225-V + Intel Wi-Fi 6 AX210/AX211/AX411 OS: Ubuntu 23.10, Kernel: 6.7.0-060700-generic (x86_64), Desktop: GNOME Shell 45.2, Display Server: X Server 1.21.1.7, Display Driver: NVIDIA 550.54.14, OpenGL: 4.6.0, OpenCL: OpenCL 3.0 CUDA 12.4.89, Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 3840x2160 c: Processor: AMD Ryzen 9 7950X 16-Core @ 5.88GHz (16 Cores / 32 Threads), Motherboard: ASUS ROG STRIX X670E-E GAMING WIFI (1905 BIOS), Chipset: AMD Device 14d8, Memory: 2 x 16GB DRAM-6000MT/s G Skill F5-6000J3038F16G, Disk: 2000GB Samsung SSD 980 PRO 2TB + Western Digital WD_BLACK SN850X 2000GB, Graphics: NVIDIA GeForce RTX 3080 10GB, Audio: NVIDIA GA102 HD Audio, Monitor: DELL U2723QE, Network: Intel I225-V + Intel Wi-Fi 6 AX210/AX211/AX411 OS: Ubuntu 23.10, Kernel: 6.7.0-060700-generic (x86_64), Desktop: GNOME Shell 45.2, Display Server: X Server 1.21.1.7, Display Driver: NVIDIA 550.54.14, OpenGL: 4.6.0, OpenCL: OpenCL 3.0 CUDA 12.4.89, Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 3840x2160 d: Processor: AMD Ryzen 9 7950X 16-Core @ 5.88GHz (16 Cores / 32 Threads), Motherboard: ASUS ROG STRIX X670E-E GAMING WIFI (1905 BIOS), Chipset: AMD Device 14d8, Memory: 2 x 16GB DRAM-6000MT/s G Skill F5-6000J3038F16G, Disk: 2000GB Samsung SSD 980 PRO 2TB + Western Digital WD_BLACK SN850X 2000GB, Graphics: NVIDIA GeForce RTX 3080 10GB, Audio: NVIDIA GA102 HD Audio, Monitor: DELL U2723QE, Network: Intel I225-V + Intel Wi-Fi 6 AX210/AX211/AX411 OS: Ubuntu 23.10, Kernel: 6.7.0-060700-generic (x86_64), Desktop: GNOME Shell 45.2, Display Server: X Server 1.21.1.7, Display Driver: NVIDIA 550.54.14, OpenGL: 4.6.0, OpenCL: OpenCL 3.0 CUDA 12.4.89, Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 3840x2160 e: Processor: AMD Ryzen 9 7950X 16-Core @ 5.88GHz (16 Cores / 32 Threads), Motherboard: ASUS ROG STRIX X670E-E GAMING WIFI (1905 BIOS), Chipset: AMD Device 14d8, Memory: 2 x 16GB DRAM-6000MT/s G Skill F5-6000J3038F16G, Disk: 2000GB Samsung SSD 980 PRO 2TB + Western Digital WD_BLACK SN850X 2000GB, Graphics: NVIDIA GeForce RTX 3080 10GB, Audio: NVIDIA GA102 HD Audio, Monitor: DELL U2723QE, Network: Intel I225-V + Intel Wi-Fi 6 AX210/AX211/AX411 OS: Ubuntu 23.10, Kernel: 6.7.0-060700-generic (x86_64), Desktop: GNOME Shell 45.2, Display Server: X Server 1.21.1.7, Display Driver: NVIDIA 550.54.14, OpenGL: 4.6.0, OpenCL: OpenCL 3.0 CUDA 12.4.89, Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 3840x2160 Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 21.60 |==================================================================== b . 21.25 |=================================================================== c . 21.28 |=================================================================== d . 21.24 |=================================================================== e . 21.31 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 369.73 |================================================================== b . 376.36 |=================================================================== c . 375.87 |=================================================================== d . 376.52 |=================================================================== e . 374.97 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 18.81 |==================================================================== b . 18.66 |=================================================================== c . 18.69 |==================================================================== d . 18.72 |==================================================================== e . 18.66 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 53.15 |=================================================================== b . 53.59 |==================================================================== c . 53.49 |==================================================================== d . 53.42 |==================================================================== e . 53.57 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 929.85 |=================================================================== b . 928.17 |=================================================================== c . 927.61 |=================================================================== d . 920.28 |================================================================== e . 926.39 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 8.5925 |================================================================== b . 8.6079 |================================================================== c . 8.6128 |================================================================== d . 8.6817 |=================================================================== e . 8.6244 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 300.87 |=================================================================== b . 301.24 |=================================================================== c . 300.11 |=================================================================== d . 301.70 |=================================================================== e . 301.17 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 3.3206 |=================================================================== b . 3.3168 |=================================================================== c . 3.3287 |=================================================================== d . 3.3113 |=================================================================== e . 3.3176 |=================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 280.62 |=================================================================== b . 280.43 |=================================================================== c . 281.03 |=================================================================== d . 280.45 |=================================================================== e . 280.54 |=================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 28.49 |==================================================================== b . 28.51 |==================================================================== c . 28.46 |==================================================================== d . 28.51 |==================================================================== e . 28.51 |==================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 195.29 |=================================================================== b . 195.96 |=================================================================== c . 195.50 |=================================================================== d . 196.02 |=================================================================== e . 195.75 |=================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 5.1149 |=================================================================== b . 5.0976 |=================================================================== c . 5.1095 |=================================================================== d . 5.0963 |=================================================================== e . 5.1030 |=================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 2379.95 |================================================================== b . 2376.57 |================================================================== c . 2385.77 |================================================================== d . 2381.27 |================================================================== e . 2359.66 |================================================================= Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 3.3518 |================================================================== b . 3.3559 |=================================================================== c . 3.3425 |================================================================== d . 3.3488 |================================================================== e . 3.3800 |=================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 1437.55 |================================================================== b . 1438.27 |================================================================== c . 1444.83 |================================================================== d . 1420.74 |================================================================= e . 1433.24 |================================================================= Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 0.6937 |================================================================== b . 0.6932 |================================================================== c . 0.6899 |================================================================== d . 0.7019 |=================================================================== e . 0.6957 |================================================================== Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 4.0959 |=================================================================== b . 4.1098 |=================================================================== c . 4.0853 |=================================================================== d . 4.0791 |================================================================== e . 4.0962 |=================================================================== Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 1900.48 |================================================================== b . 1894.22 |================================================================== c . 1905.42 |================================================================== d . 1908.61 |================================================================== e . 1900.56 |================================================================== Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 7.6108 |=================================================================== b . 7.6089 |=================================================================== c . 7.6060 |=================================================================== d . 7.5982 |=================================================================== e . 7.6006 |=================================================================== Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 131.38 |=================================================================== b . 131.41 |=================================================================== c . 131.46 |=================================================================== d . 131.60 |=================================================================== e . 131.55 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 279.44 |================================================================== b . 280.45 |=================================================================== c . 280.98 |=================================================================== d . 280.51 |=================================================================== e . 282.48 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 28.62 |==================================================================== b . 28.51 |==================================================================== c . 28.46 |==================================================================== d . 28.51 |==================================================================== e . 28.31 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 194.43 |================================================================== b . 195.40 |=================================================================== c . 196.34 |=================================================================== d . 196.37 |=================================================================== e . 195.94 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 5.1378 |=================================================================== b . 5.1120 |=================================================================== c . 5.0880 |================================================================== d . 5.0871 |================================================================== e . 5.0981 |================================================================== Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 126.34 |=================================================================== b . 126.68 |=================================================================== c . 127.03 |=================================================================== d . 126.41 |=================================================================== e . 126.73 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 63.27 |==================================================================== b . 63.12 |==================================================================== c . 62.95 |==================================================================== d . 63.26 |==================================================================== e . 63.10 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 99.00 |==================================================================== b . 99.50 |==================================================================== c . 99.41 |==================================================================== d . 99.50 |==================================================================== e . 99.10 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 10.10 |==================================================================== b . 10.05 |==================================================================== c . 10.05 |==================================================================== d . 10.05 |==================================================================== e . 10.09 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 191.70 |=================================================================== b . 192.51 |=================================================================== c . 192.61 |=================================================================== d . 192.31 |=================================================================== e . 190.79 |================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 41.71 |==================================================================== b . 41.53 |=================================================================== c . 41.52 |=================================================================== d . 41.58 |=================================================================== e . 41.92 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 117.66 |=================================================================== b . 116.35 |================================================================== c . 116.56 |================================================================== d . 115.26 |================================================================== e . 117.21 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 8.4942 |================================================================== b . 8.5902 |================================================================== c . 8.5743 |================================================================== d . 8.6715 |=================================================================== e . 8.5269 |================================================================== Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 37.65 |==================================================================== b . 37.78 |==================================================================== c . 37.76 |==================================================================== d . 37.72 |==================================================================== e . 37.61 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 212.45 |=================================================================== b . 211.75 |=================================================================== c . 211.82 |=================================================================== d . 212.09 |=================================================================== e . 212.71 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 31.57 |==================================================================== b . 31.66 |==================================================================== c . 31.66 |==================================================================== d . 31.70 |==================================================================== e . 31.56 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 31.66 |==================================================================== b . 31.57 |==================================================================== c . 31.57 |==================================================================== d . 31.53 |==================================================================== e . 31.67 |==================================================================== Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 430.43 |=================================================================== b . 431.81 |=================================================================== c . 430.89 |=================================================================== d . 430.66 |=================================================================== e . 429.15 |=================================================================== Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 18.57 |==================================================================== b . 18.51 |==================================================================== c . 18.55 |==================================================================== d . 18.56 |==================================================================== e . 18.63 |==================================================================== Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 112.87 |=================================================================== b . 112.65 |=================================================================== c . 112.25 |=================================================================== d . 112.57 |=================================================================== e . 112.53 |=================================================================== Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 8.8521 |=================================================================== b . 8.8696 |=================================================================== c . 8.9004 |=================================================================== d . 8.8748 |=================================================================== e . 8.8780 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 21.51 |==================================================================== b . 21.47 |==================================================================== c . 21.32 |=================================================================== d . 21.45 |==================================================================== e . 21.42 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 371.57 |=================================================================== b . 370.83 |================================================================== c . 374.08 |=================================================================== d . 371.95 |=================================================================== e . 372.49 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 18.60 |==================================================================== b . 18.54 |==================================================================== c . 18.61 |==================================================================== d . 18.65 |==================================================================== e . 18.62 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 53.76 |==================================================================== b . 53.93 |==================================================================== c . 53.71 |==================================================================== d . 53.60 |==================================================================== e . 53.71 |====================================================================