deepsparse 7950x AMD Ryzen 9 7950X 16-Core testing with a ASUS ROG CROSSHAIR X670E HERO (0805 BIOS) and AMD Radeon RX 6800/6800 XT / 6900 on Ubuntu 22.10 via the Phoronix Test Suite. a: Processor: AMD Ryzen 9 7950X 16-Core @ 5.88GHz (16 Cores / 32 Threads), Motherboard: ASUS ROG CROSSHAIR X670E HERO (0805 BIOS), Chipset: AMD Device 14d8, Memory: 32GB, Disk: 2048GB SOLIDIGM SSDPFKKW020X7 + 257GB Flash Drive, Graphics: AMD Radeon RX 6800/6800 XT / 6900, Audio: AMD Navi 21/23, Monitor: ASUS MG28U, Network: Intel I225-V + Intel Wi-Fi 6 AX210/AX211/AX411 OS: Ubuntu 22.10, Kernel: 5.19.0-29-generic (x86_64), Desktop: GNOME Shell 43.1, Display Server: X Server + Wayland, Vulkan: 1.3.224, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 3840x2160 bb: Processor: AMD Ryzen 9 7950X 16-Core @ 5.88GHz (16 Cores / 32 Threads), Motherboard: ASUS ROG CROSSHAIR X670E HERO (0805 BIOS), Chipset: AMD Device 14d8, Memory: 32GB, Disk: 2048GB SOLIDIGM SSDPFKKW020X7 + 257GB Flash Drive, Graphics: AMD Radeon RX 6800/6800 XT / 6900 (2575/1000MHz), Audio: AMD Navi 21/23, Monitor: ASUS MG28U, Network: Intel I225-V + Intel Wi-Fi 6 AX210/AX211/AX411 OS: Ubuntu 22.10, Kernel: 5.19.0-29-generic (x86_64), Desktop: GNOME Shell 43.1, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 22.2.1 (LLVM 15.0.2 DRM 3.47), Vulkan: 1.3.224, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 3840x2160 c: Processor: AMD Ryzen 9 7950X 16-Core @ 5.88GHz (16 Cores / 32 Threads), Motherboard: ASUS ROG CROSSHAIR X670E HERO (0805 BIOS), Chipset: AMD Device 14d8, Memory: 32GB, Disk: 2048GB SOLIDIGM SSDPFKKW020X7 + 257GB Flash Drive, Graphics: AMD Radeon RX 6800/6800 XT / 6900 (2575/1000MHz), Audio: AMD Navi 21/23, Monitor: ASUS MG28U, Network: Intel I225-V + Intel Wi-Fi 6 AX210/AX211/AX411 OS: Ubuntu 22.10, Kernel: 5.19.0-29-generic (x86_64), Desktop: GNOME Shell 43.1, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 22.2.1 (LLVM 15.0.2 DRM 3.47), Vulkan: 1.3.224, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 3840x2160 d: Processor: AMD Ryzen 9 7950X 16-Core @ 5.88GHz (16 Cores / 32 Threads), Motherboard: ASUS ROG CROSSHAIR X670E HERO (0805 BIOS), Chipset: AMD Device 14d8, Memory: 32GB, Disk: 2048GB SOLIDIGM SSDPFKKW020X7 + 257GB Flash Drive, Graphics: AMD Radeon RX 6800/6800 XT / 6900 (2575/1000MHz), Audio: AMD Navi 21/23, Monitor: ASUS MG28U, Network: Intel I225-V + Intel Wi-Fi 6 AX210/AX211/AX411 OS: Ubuntu 22.10, Kernel: 5.19.0-29-generic (x86_64), Desktop: GNOME Shell 43.1, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 22.2.1 (LLVM 15.0.2 DRM 3.47), Vulkan: 1.3.224, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 3840x2160 Neural Magic DeepSparse 1.3.2 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a .. 12.70 |=================================================================== bb . 12.64 |=================================================================== c .. 12.59 |================================================================== d .. 12.64 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a .. 627.44 |================================================================= bb . 630.52 |================================================================== c .. 632.93 |================================================================== d .. 630.36 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a .. 11.76 |=================================================================== bb . 11.72 |=================================================================== c .. 11.69 |=================================================================== d .. 11.72 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a .. 85.03 |=================================================================== bb . 85.33 |=================================================================== c .. 85.51 |=================================================================== d .. 85.32 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a .. 447.31 |================================================================== bb . 445.89 |================================================================== c .. 446.65 |================================================================== d .. 446.73 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a .. 17.87 |=================================================================== bb . 17.93 |=================================================================== c .. 17.90 |=================================================================== d .. 17.89 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a .. 163.77 |================================================================== bb . 163.25 |================================================================== c .. 162.78 |================================================================= d .. 164.27 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a .. 6.1019 |================================================================== bb . 6.1208 |================================================================== c .. 6.1393 |================================================================== d .. 6.0829 |================================================================= Neural Magic DeepSparse 1.3.2 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a .. 101.26 |================================================================== bb . 101.57 |================================================================== c .. 100.75 |================================================================= d .. 100.93 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a .. 78.98 |=================================================================== bb . 78.74 |================================================================== c .. 79.38 |=================================================================== d .. 79.24 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a .. 64.84 |=================================================================== bb . 64.16 |================================================================== c .. 64.39 |=================================================================== d .. 64.44 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a .. 15.42 |================================================================== bb . 15.58 |=================================================================== c .. 15.52 |=================================================================== d .. 15.51 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a .. 116.66 |================================================================= bb . 117.61 |================================================================== c .. 115.99 |================================================================= d .. 116.06 |================================================================= Neural Magic DeepSparse 1.3.2 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a .. 68.51 |=================================================================== bb . 67.95 |================================================================== c .. 68.93 |=================================================================== d .. 68.88 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a .. 88.72 |=================================================================== bb . 88.80 |=================================================================== c .. 89.05 |=================================================================== d .. 88.95 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a .. 11.25 |=================================================================== bb . 11.25 |=================================================================== c .. 11.21 |=================================================================== d .. 11.23 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a .. 298.27 |================================================================== bb . 298.34 |================================================================== c .. 298.62 |================================================================== d .. 297.80 |================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a .. 26.81 |=================================================================== bb . 26.80 |=================================================================== c .. 26.78 |=================================================================== d .. 26.85 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a .. 204.37 |================================================================== bb . 204.04 |================================================================== c .. 204.66 |================================================================== d .. 204.44 |================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a .. 4.8880 |================================================================== bb . 4.8960 |================================================================== c .. 4.8812 |================================================================== d .. 4.8865 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a .. 184.90 |================================================================== bb . 184.42 |================================================================== c .. 182.96 |================================================================= d .. 182.03 |================================================================= Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a .. 43.25 |================================================================== bb . 43.37 |================================================================== c .. 43.71 |=================================================================== d .. 43.94 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a .. 116.74 |================================================================== bb . 117.54 |================================================================== c .. 116.61 |================================================================= d .. 117.53 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a .. 8.5625 |================================================================== bb . 8.5043 |================================================================= c .. 8.5719 |================================================================== d .. 8.5044 |================================================================= Neural Magic DeepSparse 1.3.2 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a .. 39.76 |=================================================================== bb . 39.64 |=================================================================== c .. 39.87 |=================================================================== d .. 39.85 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a .. 201.01 |================================================================== bb . 201.36 |================================================================== c .. 200.48 |================================================================== d .. 200.53 |================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a .. 33.91 |=================================================================== bb . 33.89 |=================================================================== c .. 33.90 |=================================================================== d .. 33.87 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a .. 29.48 |=================================================================== bb . 29.49 |=================================================================== c .. 29.49 |=================================================================== d .. 29.51 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a .. 96.06 |=================================================================== bb . 95.57 |=================================================================== c .. 96.28 |=================================================================== d .. 95.61 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a .. 83.27 |=================================================================== bb . 83.70 |=================================================================== c .. 83.08 |=================================================================== d .. 83.66 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a .. 61.73 |=================================================================== bb . 61.14 |================================================================== c .. 61.90 |=================================================================== d .. 61.93 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a .. 16.20 |================================================================== bb . 16.35 |=================================================================== c .. 16.15 |================================================================== d .. 16.14 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a .. 12.59 |=================================================================== bb . 12.63 |=================================================================== c .. 12.63 |=================================================================== d .. 12.59 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a .. 633.18 |================================================================== bb . 630.80 |================================================================== c .. 631.30 |================================================================== d .. 632.16 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a .. 11.72 |=================================================================== bb . 11.71 |=================================================================== c .. 11.74 |=================================================================== d .. 11.73 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a .. 85.35 |=================================================================== bb . 85.38 |=================================================================== c .. 85.18 |=================================================================== d .. 85.22 |===================================================================