Intel Core i9-14900K testing with a ASUS PRIME Z790-P WIFI (1402 BIOS) and ASUS Intel RPL-S 31GB on Ubuntu 23.10 via the Phoronix Test Suite.
Compare your own system(s) to this result file with the
Phoronix Test Suite by running the command:
phoronix-test-suite benchmark 2403152-PTS-DEEPSPAR00 deepsparse 1.7 raptor lake - Phoronix Test Suite deepsparse 1.7 raptor lake Intel Core i9-14900K testing with a ASUS PRIME Z790-P WIFI (1402 BIOS) and ASUS Intel RPL-S 31GB on Ubuntu 23.10 via the Phoronix Test Suite.
HTML result view exported from: https://openbenchmarking.org/result/2403152-PTS-DEEPSPAR00&grs&export=txt&sro .
deepsparse 1.7 raptor lake Processor Motherboard Chipset Memory Disk Graphics Audio Monitor OS Kernel Desktop Display Server OpenGL Compiler File-System Screen Resolution a b c d e Intel Core i9-14900K @ 5.70GHz (24 Cores / 32 Threads) ASUS PRIME Z790-P WIFI (1402 BIOS) Intel Device 7a27 2 x 16GB DRAM-6000MT/s Corsair CMK32GX5M2B6000C36 Western Digital WD_BLACK SN850X 1000GB ASUS Intel RPL-S 31GB (1650MHz) Realtek ALC897 ASUS VP28U Ubuntu 23.10 6.8.0-phx (x86_64) GNOME Shell 45.1 X Server 1.21.1.7 4.6 Mesa 24.0~git2312240600.c05261~oibaf~m (git-c05261a 2023-12-24 mantic-oibaf-ppa) GCC 13.2.0 ext4 3840x2160 OpenBenchmarking.org Kernel Details - Transparent Huge Pages: madvise Processor Details - Scaling Governor: intel_pstate powersave (EPP: performance) - CPU Microcode: 0x122 - Thermald 2.5.4 Python Details - Python 3.11.6 Security Details - gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + reg_file_data_sampling: Mitigation of Clear Register File + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected
deepsparse 1.7 raptor lake deepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Stream deepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Stream deepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Stream deepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Stream deepsparse: Llama2 Chat 7b Quantized - Asynchronous Multi-Stream deepsparse: Llama2 Chat 7b Quantized - Asynchronous Multi-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Stream deepsparse: ResNet-50, Baseline - Synchronous Single-Stream deepsparse: ResNet-50, Baseline - Synchronous Single-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Stream deepsparse: ResNet-50, Baseline - Asynchronous Multi-Stream deepsparse: ResNet-50, Baseline - Asynchronous Multi-Stream deepsparse: Llama2 Chat 7b Quantized - Synchronous Single-Stream deepsparse: Llama2 Chat 7b Quantized - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Stream a b c d e 975.8077 1.0225 3.4607 1152.2519 7.6693 518.2458 20.0825 49.7864 72.9891 13.6991 374.6573 10.6773 8.0391 124.3319 104.1188 9.6041 27.9745 105.8800 142.9151 9.4444 324.5630 3.0776 7.9967 124.9918 269.9726 14.8156 81.4912 49.0714 379.7141 10.5338 9.1560 436.2483 67.2624 14.8628 20.3301 196.6202 7.9348 125.9194 27.9492 143.0378 88.8680 11.2508 71.6975 55.7746 941.7863 1.0591 3.4488 1156.1766 7.7518 512.6893 19.6607 50.8544 71.4267 13.9983 368.5855 10.8516 7.8904 126.6727 104.392 9.5789 27.7941 106.2107 143.8358 9.4148 327.1912 3.0531 7.9282 126.0684 268.0825 14.9195 81.5021 49.066 379.1422 10.5495 9.199 434.2205 67.6241 14.7838 20.2407 197.4838 7.9643 125.4487 27.9208 143.1866 88.8701 11.2506 71.7457 55.7375 927.7726 1.0746 3.53 1129.727 7.5635 525.3862 19.6462 50.892 71.4383 13.9961 373.1233 10.7197 7.9405 125.8707 105.1934 9.5059 27.9005 105.4393 143.2884 9.4838 326.888 3.056 7.9467 125.7691 267.9952 14.9245 81.5086 49.0612 378.2626 10.5739 9.1596 436.1158 67.3817 14.8369 20.2619 197.2772 7.9518 125.6334 27.8803 143.3914 88.9381 11.242 71.7627 55.7242 1011.7146 0.9856 3.4601 1152.7523 7.6435 520.0115 19.8227 50.4405 72.0667 13.8746 375.6293 10.6496 7.9734 125.3512 105.5378 9.4750 27.9110 106.7958 143.2367 9.3639 327.2907 3.0521 7.9961 125.0031 269.6208 14.8348 81.6823 48.9564 378.1676 10.5769 9.1642 435.8541 67.4110 14.8305 20.2703 197.1962 7.9571 125.5685 27.9572 142.9982 88.9280 11.2433 71.7605 55.7256 994.7677 1.0024 3.4336 1161.2978 7.6128 521.9992 19.9587 50.0942 71.9383 13.8988 368.2256 10.8623 7.9248 126.1129 104.5183 9.5673 27.6188 106.6262 144.7457 9.3782 328.5605 3.0404 7.9452 125.7981 268.5519 14.8935 81.1745 49.2619 377.3639 10.5993 9.2062 433.8822 67.3195 14.8504 20.2315 197.5747 7.9673 125.4108 27.8453 143.5701 88.978 11.2371 71.7753 55.7142 OpenBenchmarking.org
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d e 200 400 600 800 1000 SE +/- 10.53, N = 5 SE +/- 1.14, N = 3 975.81 941.79 927.77 1011.71 994.77
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d e 0.2418 0.4836 0.7254 0.9672 1.209 SE +/- 0.0112, N = 5 SE +/- 0.0010, N = 3 1.0225 1.0591 1.0746 0.9856 1.0024
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d e 0.7943 1.5886 2.3829 3.1772 3.9715 SE +/- 0.0108, N = 3 SE +/- 0.0320, N = 3 3.4607 3.4488 3.5300 3.4601 3.4336
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d e 200 400 600 800 1000 SE +/- 3.50, N = 3 SE +/- 10.56, N = 3 1152.25 1156.18 1129.73 1152.75 1161.30
Neural Magic DeepSparse Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream a b c d e 2 4 6 8 10 SE +/- 0.0698, N = 3 SE +/- 0.0734, N = 3 7.6693 7.7518 7.5635 7.6435 7.6128
Neural Magic DeepSparse Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream a b c d e 110 220 330 440 550 SE +/- 4.73, N = 3 SE +/- 4.93, N = 3 518.25 512.69 525.39 520.01 522.00
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream a b c d e 5 10 15 20 25 SE +/- 0.06, N = 3 SE +/- 0.09, N = 3 20.08 19.66 19.65 19.82 19.96
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream a b c d e 11 22 33 44 55 SE +/- 0.14, N = 3 SE +/- 0.23, N = 3 49.79 50.85 50.89 50.44 50.09
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream a b c d e 16 32 48 64 80 SE +/- 0.25, N = 3 SE +/- 0.34, N = 3 72.99 71.43 71.44 72.07 71.94
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream a b c d e 4 8 12 16 20 SE +/- 0.05, N = 3 SE +/- 0.06, N = 3 13.70 14.00 14.00 13.87 13.90
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream a b c d e 80 160 240 320 400 SE +/- 3.10, N = 3 SE +/- 3.00, N = 3 374.66 368.59 373.12 375.63 368.23
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream a b c d e 3 6 9 12 15 SE +/- 0.09, N = 3 SE +/- 0.09, N = 3 10.68 10.85 10.72 10.65 10.86
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream a b c d e 2 4 6 8 10 SE +/- 0.0189, N = 3 SE +/- 0.0250, N = 3 8.0391 7.8904 7.9405 7.9734 7.9248
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream a b c d e 30 60 90 120 150 SE +/- 0.29, N = 3 SE +/- 0.39, N = 3 124.33 126.67 125.87 125.35 126.11
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream a b c d e 20 40 60 80 100 SE +/- 0.22, N = 3 SE +/- 0.16, N = 3 104.12 104.39 105.19 105.54 104.52
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream a b c d e 3 6 9 12 15 SE +/- 0.0203, N = 3 SE +/- 0.0143, N = 3 9.6041 9.5789 9.5059 9.4750 9.5673
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream a b c d e 7 14 21 28 35 SE +/- 0.03, N = 3 SE +/- 0.04, N = 3 27.97 27.79 27.90 27.91 27.62
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream a b c d e 20 40 60 80 100 SE +/- 0.24, N = 3 SE +/- 0.61, N = 3 105.88 106.21 105.44 106.80 106.63
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream a b c d e 30 60 90 120 150 SE +/- 0.17, N = 3 SE +/- 0.22, N = 3 142.92 143.84 143.29 143.24 144.75
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream a b c d e 3 6 9 12 15 SE +/- 0.0212, N = 3 SE +/- 0.0538, N = 3 9.4444 9.4148 9.4838 9.3639 9.3782
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d e 70 140 210 280 350 SE +/- 0.47, N = 3 SE +/- 0.60, N = 3 324.56 327.19 326.89 327.29 328.56
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d e 0.6925 1.385 2.0775 2.77 3.4625 SE +/- 0.0044, N = 3 SE +/- 0.0056, N = 3 3.0776 3.0531 3.0560 3.0521 3.0404
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream a b c d e 2 4 6 8 10 SE +/- 0.0195, N = 3 SE +/- 0.0415, N = 3 7.9967 7.9282 7.9467 7.9961 7.9452
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream a b c d e 30 60 90 120 150 SE +/- 0.30, N = 3 SE +/- 0.65, N = 3 124.99 126.07 125.77 125.00 125.80
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream a b c d e 60 120 180 240 300 SE +/- 1.11, N = 3 SE +/- 0.87, N = 3 269.97 268.08 268.00 269.62 268.55
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream a b c d e 4 8 12 16 20 SE +/- 0.06, N = 3 SE +/- 0.05, N = 3 14.82 14.92 14.92 14.83 14.89
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream a b c d e 20 40 60 80 100 SE +/- 0.08, N = 3 SE +/- 0.22, N = 3 81.49 81.50 81.51 81.68 81.17
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream a b c d e 11 22 33 44 55 SE +/- 0.05, N = 3 SE +/- 0.13, N = 3 49.07 49.07 49.06 48.96 49.26
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream a b c d e 80 160 240 320 400 SE +/- 1.12, N = 3 SE +/- 1.11, N = 3 379.71 379.14 378.26 378.17 377.36
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream a b c d e 3 6 9 12 15 SE +/- 0.03, N = 3 SE +/- 0.03, N = 3 10.53 10.55 10.57 10.58 10.60
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d e 3 6 9 12 15 SE +/- 0.0346, N = 3 SE +/- 0.0105, N = 3 9.1560 9.1990 9.1596 9.1642 9.2062
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d e 90 180 270 360 450 SE +/- 1.64, N = 3 SE +/- 0.49, N = 3 436.25 434.22 436.12 435.85 433.88
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d e 15 30 45 60 75 SE +/- 0.03, N = 3 SE +/- 0.14, N = 3 67.26 67.62 67.38 67.41 67.32
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d e 4 8 12 16 20 SE +/- 0.01, N = 3 SE +/- 0.03, N = 3 14.86 14.78 14.84 14.83 14.85
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d e 5 10 15 20 25 SE +/- 0.01, N = 3 SE +/- 0.02, N = 3 20.33 20.24 20.26 20.27 20.23
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d e 40 80 120 160 200 SE +/- 0.14, N = 3 SE +/- 0.19, N = 3 196.62 197.48 197.28 197.20 197.57
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d e 2 4 6 8 10 SE +/- 0.0106, N = 3 SE +/- 0.0196, N = 3 7.9348 7.9643 7.9518 7.9571 7.9673
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d e 30 60 90 120 150 SE +/- 0.17, N = 3 SE +/- 0.31, N = 3 125.92 125.45 125.63 125.57 125.41
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream a b c d e 7 14 21 28 35 SE +/- 0.05, N = 3 SE +/- 0.01, N = 3 27.95 27.92 27.88 27.96 27.85
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream a b c d e 30 60 90 120 150 SE +/- 0.27, N = 3 SE +/- 0.03, N = 3 143.04 143.19 143.39 143.00 143.57
Neural Magic DeepSparse Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream a b c d e 20 40 60 80 100 SE +/- 0.03, N = 3 SE +/- 0.02, N = 3 88.87 88.87 88.94 88.93 88.98
Neural Magic DeepSparse Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream a b c d e 3 6 9 12 15 SE +/- 0.00, N = 3 SE +/- 0.00, N = 3 11.25 11.25 11.24 11.24 11.24
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d e 16 32 48 64 80 SE +/- 0.04, N = 3 SE +/- 0.03, N = 3 71.70 71.75 71.76 71.76 71.78
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d e 13 26 39 52 65 SE +/- 0.03, N = 3 SE +/- 0.02, N = 3 55.77 55.74 55.72 55.73 55.71
Phoronix Test Suite v10.8.4