Intel Core i9-14900K testing with a ASUS PRIME Z790-P WIFI (1402 BIOS) and ASUS Intel RPL-S 31GB on Ubuntu 23.10 via the Phoronix Test Suite.
Compare your own system(s) to this result file with the
Phoronix Test Suite by running the command:
phoronix-test-suite benchmark 2403152-PTS-DEEPSPAR00 deepsparse 1.7 raptor lake - Phoronix Test Suite deepsparse 1.7 raptor lake Intel Core i9-14900K testing with a ASUS PRIME Z790-P WIFI (1402 BIOS) and ASUS Intel RPL-S 31GB on Ubuntu 23.10 via the Phoronix Test Suite.
HTML result view exported from: https://openbenchmarking.org/result/2403152-PTS-DEEPSPAR00&sor&grw .
deepsparse 1.7 raptor lake Processor Motherboard Chipset Memory Disk Graphics Audio Monitor OS Kernel Desktop Display Server OpenGL Compiler File-System Screen Resolution a b c d e Intel Core i9-14900K @ 5.70GHz (24 Cores / 32 Threads) ASUS PRIME Z790-P WIFI (1402 BIOS) Intel Device 7a27 2 x 16GB DRAM-6000MT/s Corsair CMK32GX5M2B6000C36 Western Digital WD_BLACK SN850X 1000GB ASUS Intel RPL-S 31GB (1650MHz) Realtek ALC897 ASUS VP28U Ubuntu 23.10 6.8.0-phx (x86_64) GNOME Shell 45.1 X Server 1.21.1.7 4.6 Mesa 24.0~git2312240600.c05261~oibaf~m (git-c05261a 2023-12-24 mantic-oibaf-ppa) GCC 13.2.0 ext4 3840x2160 OpenBenchmarking.org Kernel Details - Transparent Huge Pages: madvise Processor Details - Scaling Governor: intel_pstate powersave (EPP: performance) - CPU Microcode: 0x122 - Thermald 2.5.4 Python Details - Python 3.11.6 Security Details - gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + reg_file_data_sampling: Mitigation of Clear Register File + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected
deepsparse 1.7 raptor lake deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Stream deepsparse: ResNet-50, Baseline - Asynchronous Multi-Stream deepsparse: ResNet-50, Baseline - Asynchronous Multi-Stream deepsparse: ResNet-50, Baseline - Synchronous Single-Stream deepsparse: ResNet-50, Baseline - Synchronous Single-Stream deepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Stream deepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Stream deepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Stream deepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Stream deepsparse: Llama2 Chat 7b Quantized - Asynchronous Multi-Stream deepsparse: Llama2 Chat 7b Quantized - Asynchronous Multi-Stream deepsparse: Llama2 Chat 7b Quantized - Synchronous Single-Stream deepsparse: Llama2 Chat 7b Quantized - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream a b c d e 10.6773 374.6573 9.6041 104.1188 436.2483 9.1560 324.5630 3.0776 143.0378 27.9492 124.9918 7.9967 1152.2519 3.4607 975.8077 1.0225 7.6693 518.2458 11.2508 88.8680 142.9151 27.9745 124.3319 8.0391 71.6975 55.7746 67.2624 14.8628 81.4912 49.0714 49.7864 20.0825 14.8156 269.9726 13.6991 72.9891 196.6202 20.3301 125.9194 7.9348 10.5338 379.7141 9.4444 105.8800 10.8516 368.5855 9.5789 104.392 434.2205 9.199 327.1912 3.0531 143.1866 27.9208 126.0684 7.9282 1156.1766 3.4488 941.7863 1.0591 7.7518 512.6893 11.2506 88.8701 143.8358 27.7941 126.6727 7.8904 71.7457 55.7375 67.6241 14.7838 81.5021 49.066 50.8544 19.6607 14.9195 268.0825 13.9983 71.4267 197.4838 20.2407 125.4487 7.9643 10.5495 379.1422 9.4148 106.2107 10.7197 373.1233 9.5059 105.1934 436.1158 9.1596 326.888 3.056 143.3914 27.8803 125.7691 7.9467 1129.727 3.53 927.7726 1.0746 7.5635 525.3862 11.242 88.9381 143.2884 27.9005 125.8707 7.9405 71.7627 55.7242 67.3817 14.8369 81.5086 49.0612 50.892 19.6462 14.9245 267.9952 13.9961 71.4383 197.2772 20.2619 125.6334 7.9518 10.5739 378.2626 9.4838 105.4393 10.6496 375.6293 9.4750 105.5378 435.8541 9.1642 327.2907 3.0521 142.9982 27.9572 125.0031 7.9961 1152.7523 3.4601 1011.7146 0.9856 7.6435 520.0115 11.2433 88.9280 143.2367 27.9110 125.3512 7.9734 71.7605 55.7256 67.4110 14.8305 81.6823 48.9564 50.4405 19.8227 14.8348 269.6208 13.8746 72.0667 197.1962 20.2703 125.5685 7.9571 10.5769 378.1676 9.3639 106.7958 10.8623 368.2256 9.5673 104.5183 433.8822 9.2062 328.5605 3.0404 143.5701 27.8453 125.7981 7.9452 1161.2978 3.4336 994.7677 1.0024 7.6128 521.9992 11.2371 88.978 144.7457 27.6188 126.1129 7.9248 71.7753 55.7142 67.3195 14.8504 81.1745 49.2619 50.0942 19.9587 14.8935 268.5519 13.8988 71.9383 197.5747 20.2315 125.4108 7.9673 10.5993 377.3639 9.3782 106.6262 OpenBenchmarking.org
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream e b c a d 3 6 9 12 15 SE +/- 0.09, N = 3 SE +/- 0.09, N = 3 10.86 10.85 10.72 10.68 10.65
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream e b c a d 80 160 240 320 400 SE +/- 3.10, N = 3 SE +/- 3.00, N = 3 368.23 368.59 373.12 374.66 375.63
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream a b e c d 3 6 9 12 15 SE +/- 0.0203, N = 3 SE +/- 0.0143, N = 3 9.6041 9.5789 9.5673 9.5059 9.4750
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream a b e c d 20 40 60 80 100 SE +/- 0.22, N = 3 SE +/- 0.16, N = 3 104.12 104.39 104.52 105.19 105.54
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream a c d b e 90 180 270 360 450 SE +/- 1.64, N = 3 SE +/- 0.49, N = 3 436.25 436.12 435.85 434.22 433.88
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream a c d b e 3 6 9 12 15 SE +/- 0.0346, N = 3 SE +/- 0.0105, N = 3 9.1560 9.1596 9.1642 9.1990 9.2062
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream e d b c a 70 140 210 280 350 SE +/- 0.60, N = 3 SE +/- 0.47, N = 3 328.56 327.29 327.19 326.89 324.56
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream e d b c a 0.6925 1.385 2.0775 2.77 3.4625 SE +/- 0.0056, N = 3 SE +/- 0.0044, N = 3 3.0404 3.0521 3.0531 3.0560 3.0776
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream e c b a d 30 60 90 120 150 SE +/- 0.27, N = 3 SE +/- 0.03, N = 3 143.57 143.39 143.19 143.04 143.00
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream e c b a d 7 14 21 28 35 SE +/- 0.05, N = 3 SE +/- 0.01, N = 3 27.85 27.88 27.92 27.95 27.96
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream b e c d a 30 60 90 120 150 SE +/- 0.65, N = 3 SE +/- 0.30, N = 3 126.07 125.80 125.77 125.00 124.99
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream b e c d a 2 4 6 8 10 SE +/- 0.0415, N = 3 SE +/- 0.0195, N = 3 7.9282 7.9452 7.9467 7.9961 7.9967
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream e b d a c 200 400 600 800 1000 SE +/- 10.56, N = 3 SE +/- 3.50, N = 3 1161.30 1156.18 1152.75 1152.25 1129.73
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream e b d a c 0.7943 1.5886 2.3829 3.1772 3.9715 SE +/- 0.0320, N = 3 SE +/- 0.0108, N = 3 3.4336 3.4488 3.4601 3.4607 3.5300
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream d e a b c 200 400 600 800 1000 SE +/- 1.14, N = 3 SE +/- 10.53, N = 5 1011.71 994.77 975.81 941.79 927.77
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream d e a b c 0.2418 0.4836 0.7254 0.9672 1.209 SE +/- 0.0010, N = 3 SE +/- 0.0112, N = 5 0.9856 1.0024 1.0225 1.0591 1.0746
Neural Magic DeepSparse Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream b a d e c 2 4 6 8 10 SE +/- 0.0698, N = 3 SE +/- 0.0734, N = 3 7.7518 7.6693 7.6435 7.6128 7.5635
Neural Magic DeepSparse Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream b a d e c 110 220 330 440 550 SE +/- 4.73, N = 3 SE +/- 4.93, N = 3 512.69 518.25 520.01 522.00 525.39
Neural Magic DeepSparse Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream a b d c e 3 6 9 12 15 SE +/- 0.00, N = 3 SE +/- 0.00, N = 3 11.25 11.25 11.24 11.24 11.24
Neural Magic DeepSparse Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream a b d c e 20 40 60 80 100 SE +/- 0.03, N = 3 SE +/- 0.02, N = 3 88.87 88.87 88.93 88.94 88.98
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream e b c d a 30 60 90 120 150 SE +/- 0.22, N = 3 SE +/- 0.17, N = 3 144.75 143.84 143.29 143.24 142.92
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream e b c d a 7 14 21 28 35 SE +/- 0.04, N = 3 SE +/- 0.03, N = 3 27.62 27.79 27.90 27.91 27.97
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream b e c d a 30 60 90 120 150 SE +/- 0.39, N = 3 SE +/- 0.29, N = 3 126.67 126.11 125.87 125.35 124.33
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream b e c d a 2 4 6 8 10 SE +/- 0.0250, N = 3 SE +/- 0.0189, N = 3 7.8904 7.9248 7.9405 7.9734 8.0391
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream e c d b a 16 32 48 64 80 SE +/- 0.03, N = 3 SE +/- 0.04, N = 3 71.78 71.76 71.76 71.75 71.70
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream e c d b a 13 26 39 52 65 SE +/- 0.02, N = 3 SE +/- 0.03, N = 3 55.71 55.72 55.73 55.74 55.77
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream b d c e a 15 30 45 60 75 SE +/- 0.14, N = 3 SE +/- 0.03, N = 3 67.62 67.41 67.38 67.32 67.26
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream b d c e a 4 8 12 16 20 SE +/- 0.03, N = 3 SE +/- 0.01, N = 3 14.78 14.83 14.84 14.85 14.86
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream d c b a e 20 40 60 80 100 SE +/- 0.22, N = 3 SE +/- 0.08, N = 3 81.68 81.51 81.50 81.49 81.17
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream d c b a e 11 22 33 44 55 SE +/- 0.13, N = 3 SE +/- 0.05, N = 3 48.96 49.06 49.07 49.07 49.26
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream c b d e a 11 22 33 44 55 SE +/- 0.23, N = 3 SE +/- 0.14, N = 3 50.89 50.85 50.44 50.09 49.79
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream c b d e a 5 10 15 20 25 SE +/- 0.09, N = 3 SE +/- 0.06, N = 3 19.65 19.66 19.82 19.96 20.08
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream c b e d a 4 8 12 16 20 SE +/- 0.05, N = 3 SE +/- 0.06, N = 3 14.92 14.92 14.89 14.83 14.82
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream c b e d a 60 120 180 240 300 SE +/- 0.87, N = 3 SE +/- 1.11, N = 3 268.00 268.08 268.55 269.62 269.97
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream b c e d a 4 8 12 16 20 SE +/- 0.06, N = 3 SE +/- 0.05, N = 3 14.00 14.00 13.90 13.87 13.70
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream b c e d a 16 32 48 64 80 SE +/- 0.34, N = 3 SE +/- 0.25, N = 3 71.43 71.44 71.94 72.07 72.99
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream e b c d a 40 80 120 160 200 SE +/- 0.19, N = 3 SE +/- 0.14, N = 3 197.57 197.48 197.28 197.20 196.62
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream e b c d a 5 10 15 20 25 SE +/- 0.02, N = 3 SE +/- 0.01, N = 3 20.23 20.24 20.26 20.27 20.33
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream a c d b e 30 60 90 120 150 SE +/- 0.17, N = 3 SE +/- 0.31, N = 3 125.92 125.63 125.57 125.45 125.41
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream a c d b e 2 4 6 8 10 SE +/- 0.0106, N = 3 SE +/- 0.0196, N = 3 7.9348 7.9518 7.9571 7.9643 7.9673
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream e d c b a 3 6 9 12 15 SE +/- 0.03, N = 3 SE +/- 0.03, N = 3 10.60 10.58 10.57 10.55 10.53
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream e d c b a 80 160 240 320 400 SE +/- 1.11, N = 3 SE +/- 1.12, N = 3 377.36 378.17 378.26 379.14 379.71
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream c a b e d 3 6 9 12 15 SE +/- 0.0212, N = 3 SE +/- 0.0538, N = 3 9.4838 9.4444 9.4148 9.3782 9.3639
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream c a b e d 20 40 60 80 100 SE +/- 0.24, N = 3 SE +/- 0.61, N = 3 105.44 105.88 106.21 106.63 106.80
Phoronix Test Suite v10.8.4