ds AMD Ryzen 9 7950X 16-Core testing with a ASUS ROG STRIX X670E-E GAMING WIFI (1416 BIOS) and NVIDIA NV174 8GB on Ubuntu 23.10 via the Phoronix Test Suite.
HTML result view exported from: https://openbenchmarking.org/result/2312110-PTS-DS58174320&grr&sor .
ds Processor Motherboard Chipset Memory Disk Graphics Audio Monitor Network OS Kernel Desktop Display Server Display Driver OpenGL Compiler File-System Screen Resolution a b c d AMD Ryzen 9 7950X 16-Core @ 5.88GHz (16 Cores / 32 Threads) ASUS ROG STRIX X670E-E GAMING WIFI (1416 BIOS) AMD Device 14d8 32GB 2000GB Samsung SSD 980 PRO 2TB + 4001GB Western Digital WD_BLACK SN850X 4000GB NVIDIA NV174 8GB NVIDIA GA104 HD Audio DELL U2723QE Intel I225-V + Intel Wi-Fi 6 AX210/AX211/AX411 Ubuntu 23.10 6.7.0-060700rc2daily20231127-generic (x86_64) GNOME Shell 45.1 X Server 1.21.1.7 + Wayland nouveau 4.3 Mesa 24.0~git2311260600.945288~oibaf~m (git-945288f 2023-11-26 mantic-oibaf-ppa) GCC 13.2.0 + LLVM 16.0.6 ext4 3840x2160 OpenBenchmarking.org Kernel Details - Transparent Huge Pages: madvise Processor Details - Scaling Governor: amd-pstate-epp powersave (EPP: balance_performance) - CPU Microcode: 0xa601203 Python Details - Python 3.11.6 Security Details - gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_rstack_overflow: Vulnerable: Safe RET no microcode + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected
ds deepsparse: BERT-Large, NLP Question Answering - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream deepsparse: ResNet-50, Baseline - Asynchronous Multi-Stream deepsparse: ResNet-50, Baseline - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO - Asynchronous Multi-Stream deepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Stream deepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Stream deepsparse: ResNet-50, Baseline - Synchronous Single-Stream deepsparse: ResNet-50, Baseline - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream deepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Stream deepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Stream a b c d 47.3214 21.1279 272.1665 29.3517 3.3312 299.9092 8.1243 983.3126 17.2878 462.4186 350.7492 22.7389 9.0307 110.6228 349.9757 22.7592 51.0955 19.5685 51.1656 19.5417 204.8572 38.9835 30.8613 32.3888 39.2065 203.9666 8.5884 116.3673 27.0091 296.0378 59.8641 133.5376 62.6288 127.7126 3.4471 2314.2272 9.8076 101.8661 27.0107 296.0322 9.6870 103.1665 5.0670 197.0798 5.0217 198.8967 0.7695 1295.8078 47.2956 21.1400 271.9143 29.3894 3.3138 301.5141 8.0835 988.2981 17.2779 462.6802 351.5559 22.6899 9.0802 110.0314 351.5076 22.7193 51.1690 19.5391 51.1975 19.5292 205.5998 38.8700 30.8326 32.4189 39.2941 203.4893 8.5729 116.5816 27.0574 295.5210 60.0283 133.1707 62.8602 127.2462 3.4442 2315.5240 9.8180 101.7672 27.0344 295.7606 9.6883 103.1635 5.0320 198.4779 5.0331 198.4507 0.7698 1295.2310 47.3165 21.1302 271.7780 29.3969 3.3173 301.1738 8.0963 986.6769 17.3189 461.5344 351.4011 22.7000 9.0770 110.0550 351.4431 22.6850 51.1551 19.5454 51.1932 19.5308 204.5755 39.0428 30.8484 32.4022 39.2164 203.9185 8.5631 116.7136 27.0386 295.7285 60.5733 132.0251 62.7323 127.4662 3.4420 2317.2555 9.8068 101.8841 27.0809 295.2366 9.6736 103.3168 5.0382 198.2479 5.0250 198.7653 0.7620 1308.7670 47.0684 21.2410 272.1527 29.3591 3.3119 301.6484 8.0908 987.4091 17.2882 462.4074 352.4834 22.6176 9.0414 110.4956 351.4079 22.7096 51.0971 19.5678 51.2514 19.5097 204.2205 39.1305 30.8462 32.4054 39.2783 203.6193 8.5234 117.2571 27.0742 295.3355 60.2946 132.5933 62.4691 128.0169 3.4780 2293.2480 9.8453 101.4802 27.0539 295.5731 9.6809 103.2328 5.0356 198.3620 5.0198 198.9679 0.7672 1299.0909 OpenBenchmarking.org
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream d b c a 11 22 33 44 55 SE +/- 0.01, N = 3 SE +/- 0.09, N = 3 SE +/- 0.05, N = 3 SE +/- 0.03, N = 3 47.07 47.30 47.32 47.32
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream d b c a 5 10 15 20 25 SE +/- 0.01, N = 3 SE +/- 0.04, N = 3 SE +/- 0.02, N = 3 SE +/- 0.01, N = 3 21.24 21.14 21.13 21.13
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream c b d a 60 120 180 240 300 SE +/- 0.09, N = 3 SE +/- 0.13, N = 3 SE +/- 0.09, N = 3 SE +/- 0.01, N = 3 271.78 271.91 272.15 272.17
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream c b d a 7 14 21 28 35 SE +/- 0.01, N = 3 SE +/- 0.01, N = 3 SE +/- 0.01, N = 3 SE +/- 0.01, N = 3 29.40 29.39 29.36 29.35
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream d b c a 0.7495 1.499 2.2485 2.998 3.7475 SE +/- 0.0036, N = 3 SE +/- 0.0056, N = 3 SE +/- 0.0115, N = 3 SE +/- 0.0038, N = 3 3.3119 3.3138 3.3173 3.3312
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream d b c a 70 140 210 280 350 SE +/- 0.32, N = 3 SE +/- 0.53, N = 3 SE +/- 1.04, N = 3 SE +/- 0.35, N = 3 301.65 301.51 301.17 299.91
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream b d c a 2 4 6 8 10 SE +/- 0.0081, N = 3 SE +/- 0.0118, N = 3 SE +/- 0.0093, N = 3 SE +/- 0.0039, N = 3 8.0835 8.0908 8.0963 8.1243
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream b d c a 200 400 600 800 1000 SE +/- 0.99, N = 3 SE +/- 1.44, N = 3 SE +/- 1.16, N = 3 SE +/- 0.48, N = 3 988.30 987.41 986.68 983.31
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream b a d c 4 8 12 16 20 SE +/- 0.02, N = 3 SE +/- 0.01, N = 3 SE +/- 0.01, N = 3 SE +/- 0.01, N = 3 17.28 17.29 17.29 17.32
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream b a d c 100 200 300 400 500 SE +/- 0.43, N = 3 SE +/- 0.23, N = 3 SE +/- 0.35, N = 3 SE +/- 0.34, N = 3 462.68 462.42 462.41 461.53
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream a c b d 80 160 240 320 400 SE +/- 0.16, N = 3 SE +/- 0.75, N = 3 SE +/- 0.09, N = 3 SE +/- 0.20, N = 3 350.75 351.40 351.56 352.48
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream a c b d 5 10 15 20 25 SE +/- 0.03, N = 3 SE +/- 0.08, N = 3 SE +/- 0.03, N = 3 SE +/- 0.03, N = 3 22.74 22.70 22.69 22.62
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream a d c b 3 6 9 12 15 SE +/- 0.0337, N = 3 SE +/- 0.0640, N = 3 SE +/- 0.0179, N = 3 SE +/- 0.0759, N = 3 9.0307 9.0414 9.0770 9.0802
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream a d c b 20 40 60 80 100 SE +/- 0.42, N = 3 SE +/- 0.78, N = 3 SE +/- 0.22, N = 3 SE +/- 0.94, N = 3 110.62 110.50 110.06 110.03
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream a d c b 80 160 240 320 400 SE +/- 1.12, N = 3 SE +/- 0.17, N = 3 SE +/- 0.11, N = 3 SE +/- 0.23, N = 3 349.98 351.41 351.44 351.51
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream a b d c 5 10 15 20 25 SE +/- 0.07, N = 3 SE +/- 0.03, N = 3 SE +/- 0.01, N = 3 SE +/- 0.02, N = 3 22.76 22.72 22.71 22.69
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream a d c b 12 24 36 48 60 SE +/- 0.06, N = 3 SE +/- 0.01, N = 3 SE +/- 0.04, N = 3 SE +/- 0.02, N = 3 51.10 51.10 51.16 51.17
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream a d c b 5 10 15 20 25 SE +/- 0.02, N = 3 SE +/- 0.01, N = 3 SE +/- 0.01, N = 3 SE +/- 0.01, N = 3 19.57 19.57 19.55 19.54
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream a c b d 12 24 36 48 60 SE +/- 0.02, N = 3 SE +/- 0.07, N = 3 SE +/- 0.03, N = 3 SE +/- 0.08, N = 3 51.17 51.19 51.20 51.25
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream a c b d 5 10 15 20 25 SE +/- 0.01, N = 3 SE +/- 0.03, N = 3 SE +/- 0.01, N = 3 SE +/- 0.03, N = 3 19.54 19.53 19.53 19.51
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream d c a b 50 100 150 200 250 SE +/- 0.62, N = 3 SE +/- 0.32, N = 3 SE +/- 0.47, N = 3 SE +/- 1.77, N = 3 204.22 204.58 204.86 205.60
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream d c a b 9 18 27 36 45 SE +/- 0.13, N = 3 SE +/- 0.07, N = 3 SE +/- 0.08, N = 3 SE +/- 0.36, N = 3 39.13 39.04 38.98 38.87
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream b d c a 7 14 21 28 35 SE +/- 0.02, N = 3 SE +/- 0.04, N = 3 SE +/- 0.01, N = 3 SE +/- 0.05, N = 3 30.83 30.85 30.85 30.86
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream b d c a 8 16 24 32 40 SE +/- 0.02, N = 3 SE +/- 0.04, N = 3 SE +/- 0.01, N = 3 SE +/- 0.05, N = 3 32.42 32.41 32.40 32.39
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream a c d b 9 18 27 36 45 SE +/- 0.03, N = 3 SE +/- 0.04, N = 3 SE +/- 0.05, N = 3 SE +/- 0.01, N = 3 39.21 39.22 39.28 39.29
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream a c d b 40 80 120 160 200 SE +/- 0.15, N = 3 SE +/- 0.24, N = 3 SE +/- 0.23, N = 3 SE +/- 0.06, N = 3 203.97 203.92 203.62 203.49
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream d c b a 2 4 6 8 10 SE +/- 0.0316, N = 3 SE +/- 0.0624, N = 3 SE +/- 0.0593, N = 3 SE +/- 0.0256, N = 3 8.5234 8.5631 8.5729 8.5884
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream d c b a 30 60 90 120 150 SE +/- 0.43, N = 3 SE +/- 0.84, N = 3 SE +/- 0.80, N = 3 SE +/- 0.35, N = 3 117.26 116.71 116.58 116.37
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream a c b d 6 12 18 24 30 SE +/- 0.01, N = 3 SE +/- 0.01, N = 3 SE +/- 0.01, N = 3 SE +/- 0.01, N = 3 27.01 27.04 27.06 27.07
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream a c b d 60 120 180 240 300 SE +/- 0.11, N = 3 SE +/- 0.15, N = 3 SE +/- 0.12, N = 3 SE +/- 0.10, N = 3 296.04 295.73 295.52 295.34
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b d c 14 28 42 56 70 SE +/- 0.13, N = 3 SE +/- 0.27, N = 3 SE +/- 0.21, N = 3 SE +/- 0.21, N = 3 59.86 60.03 60.29 60.57
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b d c 30 60 90 120 150 SE +/- 0.30, N = 3 SE +/- 0.59, N = 3 SE +/- 0.43, N = 3 SE +/- 0.45, N = 3 133.54 133.17 132.59 132.03
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream d a c b 14 28 42 56 70 SE +/- 0.12, N = 3 SE +/- 0.30, N = 3 SE +/- 0.37, N = 3 SE +/- 0.28, N = 3 62.47 62.63 62.73 62.86
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream d a c b 30 60 90 120 150 SE +/- 0.24, N = 3 SE +/- 0.61, N = 3 SE +/- 0.72, N = 3 SE +/- 0.56, N = 3 128.02 127.71 127.47 127.25
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream c b a d 0.7826 1.5652 2.3478 3.1304 3.913 SE +/- 0.0064, N = 3 SE +/- 0.0106, N = 3 SE +/- 0.0021, N = 3 SE +/- 0.0122, N = 3 3.4420 3.4442 3.4471 3.4780
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream c b a d 500 1000 1500 2000 2500 SE +/- 4.26, N = 3 SE +/- 6.99, N = 3 SE +/- 1.66, N = 3 SE +/- 8.04, N = 3 2317.26 2315.52 2314.23 2293.25
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream c a b d 3 6 9 12 15 SE +/- 0.0035, N = 3 SE +/- 0.0059, N = 3 SE +/- 0.0053, N = 3 SE +/- 0.0182, N = 3 9.8068 9.8076 9.8180 9.8453
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream c a b d 20 40 60 80 100 SE +/- 0.04, N = 3 SE +/- 0.06, N = 3 SE +/- 0.06, N = 3 SE +/- 0.19, N = 3 101.88 101.87 101.77 101.48
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream a b d c 6 12 18 24 30 SE +/- 0.02, N = 3 SE +/- 0.02, N = 3 SE +/- 0.03, N = 3 SE +/- 0.02, N = 3 27.01 27.03 27.05 27.08
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream a b d c 60 120 180 240 300 SE +/- 0.22, N = 3 SE +/- 0.22, N = 3 SE +/- 0.38, N = 3 SE +/- 0.26, N = 3 296.03 295.76 295.57 295.24
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream c d a b 3 6 9 12 15 SE +/- 0.0004, N = 3 SE +/- 0.0125, N = 3 SE +/- 0.0055, N = 3 SE +/- 0.0056, N = 3 9.6736 9.6809 9.6870 9.6883
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream c d a b 20 40 60 80 100 SE +/- 0.00, N = 3 SE +/- 0.14, N = 3 SE +/- 0.06, N = 3 SE +/- 0.06, N = 3 103.32 103.23 103.17 103.16
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream b d c a 1.1401 2.2802 3.4203 4.5604 5.7005 SE +/- 0.0228, N = 3 SE +/- 0.0087, N = 3 SE +/- 0.0010, N = 3 SE +/- 0.0211, N = 3 5.0320 5.0356 5.0382 5.0670
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream b d c a 40 80 120 160 200 SE +/- 0.91, N = 3 SE +/- 0.35, N = 3 SE +/- 0.03, N = 3 SE +/- 0.82, N = 3 198.48 198.36 198.25 197.08
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream d a c b 1.1324 2.2648 3.3972 4.5296 5.662 SE +/- 0.0116, N = 3 SE +/- 0.0028, N = 3 SE +/- 0.0044, N = 3 SE +/- 0.0078, N = 3 5.0198 5.0217 5.0250 5.0331
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream d a c b 40 80 120 160 200 SE +/- 0.45, N = 3 SE +/- 0.11, N = 3 SE +/- 0.17, N = 3 SE +/- 0.31, N = 3 198.97 198.90 198.77 198.45
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream c d a b 0.1732 0.3464 0.5196 0.6928 0.866 SE +/- 0.0008, N = 3 SE +/- 0.0001, N = 3 SE +/- 0.0065, N = 3 SE +/- 0.0041, N = 3 0.7620 0.7672 0.7695 0.7698
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream c d a b 300 600 900 1200 1500 SE +/- 1.41, N = 3 SE +/- 0.19, N = 3 SE +/- 10.93, N = 3 SE +/- 7.10, N = 3 1308.77 1299.09 1295.81 1295.23
Phoronix Test Suite v10.8.5