deepsparse new AMD Ryzen 9 5900HX testing with a ASUS G513QY v1.0 (G513QY.318 BIOS) and ASUS AMD Cezanne 512MB on Ubuntu 22.10 via the Phoronix Test Suite. a: Processor: AMD Ryzen 9 5900HX @ 4.89GHz (8 Cores / 16 Threads), Motherboard: ASUS G513QY v1.0 (G513QY.318 BIOS), Chipset: AMD Renoir/Cezanne, Memory: 16GB, Disk: 512GB SAMSUNG MZVLQ512HBLU-00B00, Graphics: ASUS AMD Cezanne 512MB (2500/1000MHz), Audio: AMD Navi 21/23, Monitor: LQ156M1JW25, Network: Realtek RTL8111/8168/8411 + MEDIATEK MT7921 802.11ax PCI OS: Ubuntu 22.10, Kernel: 5.19.0-28-generic (x86_64), Desktop: GNOME Shell 43.0, Display Server: X Server 1.21.1.4 + Wayland, OpenGL: 4.6 Mesa 22.2.1 (LLVM 15.0.2 DRM 3.47), Vulkan: 1.3.224, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1080 b: Processor: AMD Ryzen 9 5900HX @ 4.89GHz (8 Cores / 16 Threads), Motherboard: ASUS G513QY v1.0 (G513QY.318 BIOS), Chipset: AMD Renoir/Cezanne, Memory: 16GB, Disk: 512GB SAMSUNG MZVLQ512HBLU-00B00, Graphics: ASUS AMD Cezanne 512MB (2500/1000MHz), Audio: AMD Navi 21/23, Monitor: LQ156M1JW25, Network: Realtek RTL8111/8168/8411 + MEDIATEK MT7921 802.11ax PCI OS: Ubuntu 22.10, Kernel: 5.19.0-28-generic (x86_64), Desktop: GNOME Shell 43.0, Display Server: X Server 1.21.1.4 + Wayland, OpenGL: 4.6 Mesa 22.2.1 (LLVM 15.0.2 DRM 3.47), Vulkan: 1.3.224, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1080 c: Processor: AMD Ryzen 9 5900HX @ 4.89GHz (8 Cores / 16 Threads), Motherboard: ASUS G513QY v1.0 (G513QY.318 BIOS), Chipset: AMD Renoir/Cezanne, Memory: 16GB, Disk: 512GB SAMSUNG MZVLQ512HBLU-00B00, Graphics: ASUS AMD Cezanne 512MB (2500/1000MHz), Audio: AMD Navi 21/23, Monitor: LQ156M1JW25, Network: Realtek RTL8111/8168/8411 + MEDIATEK MT7921 802.11ax PCI OS: Ubuntu 22.10, Kernel: 5.19.0-28-generic (x86_64), Desktop: GNOME Shell 43.0, Display Server: X Server 1.21.1.4 + Wayland, OpenGL: 4.6 Mesa 22.2.1 (LLVM 15.0.2 DRM 3.47), Vulkan: 1.3.224, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1080 Neural Magic DeepSparse 1.3.2 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 6.1849 |=================================================================== b . 6.1688 |=================================================================== c . 6.1581 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 5.8779 |=================================================================== b . 5.9081 |=================================================================== c . 5.8876 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 71.17 |==================================================================== b . 71.05 |==================================================================== c . 71.32 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 57.36 |=================================================================== b . 57.66 |==================================================================== c . 57.98 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 19.27 |=================================================================== b . 19.58 |==================================================================== c . 19.31 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 18.58 |==================================================================== b . 18.57 |==================================================================== c . 18.52 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 31.89 |==================================================================== b . 31.55 |=================================================================== c . 31.65 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 28.86 |==================================================================== b . 28.94 |==================================================================== c . 29.00 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 76.10 |==================================================================== b . 76.03 |==================================================================== c . 75.52 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 71.67 |==================================================================== b . 69.45 |================================================================== c . 70.66 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 55.51 |==================================================================== b . 54.93 |=================================================================== c . 55.01 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 47.02 |=================================================================== b . 47.42 |==================================================================== c . 47.32 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 8.5515 |=================================================================== b . 7.9379 |============================================================== c . 8.4650 |================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 8.1972 |================================================================== b . 8.3643 |=================================================================== c . 8.2039 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 26.84 |================================================================= b . 28.23 |==================================================================== c . 28.14 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 23.92 |==================================================================== b . 24.05 |==================================================================== c . 23.86 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 6.1594 |=================================================================== b . 6.1420 |================================================================== c . 6.1952 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 5.8462 |================================================================== b . 5.9750 |=================================================================== c . 5.8801 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 646.72 |=================================================================== b . 648.40 |=================================================================== c . 649.42 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 170.12 |=================================================================== b . 169.25 |=================================================================== c . 169.91 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 56.19 |==================================================================== b . 56.28 |==================================================================== c . 56.07 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 17.42 |==================================================================== b . 17.34 |==================================================================== c . 17.24 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 207.56 |=================================================================== b . 204.23 |================================================================== c . 207.02 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 53.80 |==================================================================== b . 53.83 |==================================================================== c . 54.00 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 125.38 |================================================================== b . 126.73 |=================================================================== c . 126.30 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 34.64 |==================================================================== b . 34.55 |==================================================================== c . 34.47 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 52.55 |==================================================================== b . 52.58 |==================================================================== c . 52.92 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 13.94 |================================================================== b . 14.39 |==================================================================== c . 14.15 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 72.04 |=================================================================== b . 72.81 |==================================================================== c . 72.69 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 21.26 |==================================================================== b . 21.08 |=================================================================== c . 21.13 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 467.73 |============================================================== b . 502.31 |=================================================================== c . 472.52 |=============================================================== Neural Magic DeepSparse 1.3.2 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 121.98 |=================================================================== b . 119.54 |================================================================== c . 121.90 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 148.93 |=================================================================== b . 141.70 |================================================================ c . 142.09 |================================================================ Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 41.81 |==================================================================== b . 41.58 |=================================================================== c . 41.90 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 649.39 |=================================================================== b . 651.22 |=================================================================== c . 645.61 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 171.04 |=================================================================== b . 167.36 |================================================================== c . 170.11 |===================================================================