deepsparse-54 AMD Ryzen 9 3900X 12-Core testing with a ASUS ROG STRIX X570-E GAMING (4021 BIOS) and MSI NVIDIA GeForce GTX 1050 Ti 4GB on Ubuntu 20.04 via the Phoronix Test Suite.
HTML result view exported from: https://openbenchmarking.org/result/2211300-NE-DEEPSPARS99 .
deepsparse-54 Processor Motherboard Chipset Memory Disk Graphics Audio Network OS Kernel Display Driver Vulkan Compiler File-System Screen Resolution AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX AMD Ryzen 9 3900X 12-Core @ 3.80GHz (12 Cores / 24 Threads) ASUS ROG STRIX X570-E GAMING (4021 BIOS) AMD Starship/Matisse 32GB 500GB Western Digital WDS500G3X0C-00SJG0 + Samsung SSD 970 EVO Plus 500GB MSI NVIDIA GeForce GTX 1050 Ti 4GB NVIDIA GP107GL HD Audio Realtek RTL8125 2.5GbE + Intel I211 + Intel Wi-Fi 6 AX200 Ubuntu 20.04 5.4.0-132-generic (x86_64) NVIDIA 1.1.182 GCC 9.4.0 ext4 1024x768 OpenBenchmarking.org - Transparent Huge Pages: madvise - Scaling Governor: acpi-cpufreq ondemand (Boost: Enabled) - CPU Microcode: 0x8701021 - Python 3.8.10 - itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Vulnerable + spec_store_bypass: Mitigation of SSB disabled via prctl and seccomp + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional STIBP: conditional RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected
deepsparse-54 deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Asynchronous Multi-Stream deepsparse: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Asynchronous Multi-Stream deepsparse: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Synchronous Single-Stream deepsparse: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Synchronous Single-Stream deepsparse: CV Detection,YOLOv5s COCO - Asynchronous Multi-Stream deepsparse: CV Detection,YOLOv5s COCO - Asynchronous Multi-Stream deepsparse: CV Detection,YOLOv5s COCO - Synchronous Single-Stream deepsparse: CV Detection,YOLOv5s COCO - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2 - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2 - Synchronous Single-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 10.1252 589.4118 7.8989 126.5932 36.0780 166.2700 19.4622 51.3745 56.3606 106.3857 42.6638 23.4277 113.8984 52.6262 76.9237 12.9916 82.2290 72.9129 58.0689 17.2140 41.4159 144.7796 29.2849 34.1403 10.1663 588.7073 7.9030 126.5269 OpenBenchmarking.org
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.1 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 3 6 9 12 15 SE +/- 0.05, N = 3 10.13
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.1 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 130 260 390 520 650 SE +/- 2.34, N = 3 589.41
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.1 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 2 4 6 8 10 SE +/- 0.0091, N = 3 7.8989
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.1 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 30 60 90 120 150 SE +/- 0.15, N = 3 126.59
Neural Magic DeepSparse Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.1 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 8 16 24 32 40 SE +/- 0.01, N = 3 36.08
Neural Magic DeepSparse Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.1 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 40 80 120 160 200 SE +/- 0.03, N = 3 166.27
Neural Magic DeepSparse Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.1 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 5 10 15 20 25 SE +/- 0.07, N = 3 19.46
Neural Magic DeepSparse Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.1 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 12 24 36 48 60 SE +/- 0.18, N = 3 51.37
Neural Magic DeepSparse Model: CV Detection,YOLOv5s COCO - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.1 Model: CV Detection,YOLOv5s COCO - Scenario: Asynchronous Multi-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 13 26 39 52 65 SE +/- 0.23, N = 3 56.36
Neural Magic DeepSparse Model: CV Detection,YOLOv5s COCO - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.1 Model: CV Detection,YOLOv5s COCO - Scenario: Asynchronous Multi-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 20 40 60 80 100 SE +/- 0.44, N = 3 106.39
Neural Magic DeepSparse Model: CV Detection,YOLOv5s COCO - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.1 Model: CV Detection,YOLOv5s COCO - Scenario: Synchronous Single-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 10 20 30 40 50 SE +/- 0.03, N = 3 42.66
Neural Magic DeepSparse Model: CV Detection,YOLOv5s COCO - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.1 Model: CV Detection,YOLOv5s COCO - Scenario: Synchronous Single-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 6 12 18 24 30 SE +/- 0.02, N = 3 23.43
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.1 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 30 60 90 120 150 SE +/- 0.15, N = 3 113.90
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.1 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 12 24 36 48 60 SE +/- 0.07, N = 3 52.63
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.1 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 20 40 60 80 100 SE +/- 0.15, N = 3 76.92
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.1 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 3 6 9 12 15 SE +/- 0.02, N = 3 12.99
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.1 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 20 40 60 80 100 SE +/- 0.20, N = 3 82.23
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.1 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 16 32 48 64 80 SE +/- 0.16, N = 3 72.91
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.1 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 13 26 39 52 65 SE +/- 0.04, N = 3 58.07
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.1 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 4 8 12 16 20 SE +/- 0.01, N = 3 17.21
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.1 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 9 18 27 36 45 SE +/- 0.08, N = 3 41.42
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.1 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 30 60 90 120 150 SE +/- 0.30, N = 3 144.78
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.1 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 7 14 21 28 35 SE +/- 0.01, N = 3 29.28
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.1 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 8 16 24 32 40 SE +/- 0.01, N = 3 34.14
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.1 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 3 6 9 12 15 SE +/- 0.02, N = 3 10.17
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.1 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 130 260 390 520 650 SE +/- 0.55, N = 3 588.71
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.1 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 2 4 6 8 10 SE +/- 0.0080, N = 3 7.9030
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.1 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream AMD Ryzen 9 3900X 12-Core - MSI NVIDIA GeForce GTX 30 60 90 120 150 SE +/- 0.13, N = 3 126.53
Phoronix Test Suite v10.8.4