2 x AMD EPYC 9684X 96-Core testing with a AMD Titanite_4G (RTI1007B BIOS) and ASPEED on Ubuntu 23.10 via the Phoronix Test Suite.
Compare your own system(s) to this result file with the
Phoronix Test Suite by running the command:
phoronix-test-suite benchmark 2403164-NE-DDFG2505160 ddfg - Phoronix Test Suite ddfg 2 x AMD EPYC 9684X 96-Core testing with a AMD Titanite_4G (RTI1007B BIOS) and ASPEED on Ubuntu 23.10 via the Phoronix Test Suite.
HTML result view exported from: https://openbenchmarking.org/result/2403164-NE-DDFG2505160&grw&export=pdf&sor .
ddfg Processor Motherboard Chipset Memory Disk Graphics Network OS Kernel Compiler File-System Screen Resolution a b c d 2 x AMD EPYC 9684X 96-Core @ 2.55GHz (192 Cores / 384 Threads) AMD Titanite_4G (RTI1007B BIOS) AMD Device 14a4 1520GB 3201GB Micron_7450_MTFDKCB3T2TFS + 257GB Flash Drive ASPEED Broadcom NetXtreme BCM5720 PCIe Ubuntu 23.10 6.5.0-25-generic (x86_64) GCC 13.2.0 ext4 640x480 OpenBenchmarking.org Kernel Details - Transparent Huge Pages: madvise Compiler Details - --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-13-XYspKM/gcc-13-13.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-13-XYspKM/gcc-13-13.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details - Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa10113e Python Details - Python 3.11.6 Security Details - gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_rstack_overflow: Mitigation of Safe RET + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected
ddfg deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Stream deepsparse: ResNet-50, Baseline - Asynchronous Multi-Stream deepsparse: ResNet-50, Baseline - Asynchronous Multi-Stream deepsparse: ResNet-50, Baseline - Synchronous Single-Stream deepsparse: ResNet-50, Baseline - Synchronous Single-Stream deepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Stream deepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Stream deepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Stream deepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Stream deepsparse: Llama2 Chat 7b Quantized - Asynchronous Multi-Stream deepsparse: Llama2 Chat 7b Quantized - Asynchronous Multi-Stream deepsparse: Llama2 Chat 7b Quantized - Synchronous Single-Stream deepsparse: Llama2 Chat 7b Quantized - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream primesieve: 1e12 primesieve: 1e13 svt-av1: Preset 4 - Bosphorus 4K svt-av1: Preset 8 - Bosphorus 4K svt-av1: Preset 12 - Bosphorus 4K svt-av1: Preset 13 - Bosphorus 4K svt-av1: Preset 4 - Bosphorus 1080p svt-av1: Preset 8 - Bosphorus 1080p svt-av1: Preset 12 - Bosphorus 1080p svt-av1: Preset 13 - Bosphorus 1080p a b c d 132.9523 713.5443 48.516 20.6047 5566.3853 17.2212 191.5137 5.218 1762.5286 54.3739 206.8154 4.8321 17421.4665 5.4946 807.1699 1.2364 2.9358 24242.3848 20.8303 47.9735 1763.1949 54.3534 207.6733 4.8123 798.6248 119.9052 212.1812 4.7104 1144.5034 83.6574 225.4674 4.4325 249.7234 381.0376 64.8907 15.3899 2598.2083 36.8891 68.5822 14.5734 132.9467 715.4595 48.5284 20.5991 1.164 11.789 8.557 89.277 149.069 159.218 22.964 176.674 572.349 564.413 132.9817 717.4900 48.3555 20.6727 5538.9806 17.3043 191.7714 5.2109 1758.8057 54.4869 209.7711 4.7642 17350.4640 5.5170 804.4025 1.2409 2.8607 24888.1563 20.8052 48.0337 1756.1742 54.5660 208.7235 4.7882 796.5966 120.1787 212.4945 4.7035 1142.9944 83.8482 225.4496 4.4329 249.4018 382.1237 64.8586 15.3982 2595.1763 36.9324 68.3972 14.6134 132.7968 717.2565 48.4228 20.6441 1.164 11.848 8.575 91.288 163.175 161.653 22.898 182.531 566.993 606.376 132.8792 715.8770 48.5178 20.6037 5534.2604 17.3188 192.2335 5.1985 1759.9562 54.4588 208.7406 4.7875 17333.3501 5.5224 806.2338 1.2381 2.5008 28968.9536 20.7259 48.2177 1756.5604 54.5610 208.6327 4.7905 795.0277 120.3976 212.6390 4.7002 1141.2417 83.9259 225.4415 4.4329 249.0943 382.1742 64.9222 15.3844 2595.4788 36.9216 68.5565 14.5791 133.8061 709.2588 48.3841 20.6607 1.156 11.871 8.484 90.441 160.560 162.020 23.474 184.132 568.209 588.450 133.012 714.456 48.5095 20.6072 5554.0345 17.257 194.4729 5.1383 1763.4247 54.3609 208.9767 4.7823 17400.5797 5.502 806.3219 1.2375 2.9271 24331.0626 20.4622 48.8364 1758.3058 54.505 209.1911 4.7773 797.7062 119.9993 213.5555 4.6801 1144.0054 83.7657 225.5614 4.4307 249.7754 381.1234 64.997 15.366 2592.8516 36.9564 68.7657 14.5348 132.9148 715.4294 48.379 20.6629 1.142 11.841 8.634 86.593 163.5 162.377 22.713 183.285 577.444 598.04 OpenBenchmarking.org
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream d b a c 30 60 90 120 150 SE +/- 0.21, N = 3 SE +/- 0.23, N = 3 133.01 132.98 132.95 132.88
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream a d c b 150 300 450 600 750 SE +/- 3.41, N = 3 SE +/- 1.03, N = 3 713.54 714.46 715.88 717.49
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream c a d b 11 22 33 44 55 SE +/- 0.04, N = 3 SE +/- 0.03, N = 3 48.52 48.52 48.51 48.36
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream c a d b 5 10 15 20 25 SE +/- 0.02, N = 3 SE +/- 0.01, N = 3 20.60 20.60 20.61 20.67
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream a d b c 1200 2400 3600 4800 6000 SE +/- 6.35, N = 3 SE +/- 7.14, N = 3 5566.39 5554.03 5538.98 5534.26
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream a d b c 4 8 12 16 20 SE +/- 0.02, N = 3 SE +/- 0.02, N = 3 17.22 17.26 17.30 17.32
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream d c b a 40 80 120 160 200 SE +/- 0.62, N = 3 SE +/- 0.65, N = 3 194.47 192.23 191.77 191.51
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream d c b a 1.1741 2.3482 3.5223 4.6964 5.8705 SE +/- 0.0167, N = 3 SE +/- 0.0177, N = 3 5.1383 5.1985 5.2109 5.2180
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream d a c b 400 800 1200 1600 2000 SE +/- 2.77, N = 3 SE +/- 2.58, N = 3 1763.42 1762.53 1759.96 1758.81
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream d a c b 12 24 36 48 60 SE +/- 0.08, N = 3 SE +/- 0.09, N = 3 54.36 54.37 54.46 54.49
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream b d c a 50 100 150 200 250 SE +/- 0.44, N = 3 SE +/- 0.35, N = 3 209.77 208.98 208.74 206.82
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream b d c a 1.0872 2.1744 3.2616 4.3488 5.436 SE +/- 0.0100, N = 3 SE +/- 0.0081, N = 3 4.7642 4.7823 4.7875 4.8321
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream a d b c 4K 8K 12K 16K 20K SE +/- 17.22, N = 3 SE +/- 34.07, N = 3 17421.47 17400.58 17350.46 17333.35
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream a d b c 1.2425 2.485 3.7275 4.97 6.2125 SE +/- 0.0052, N = 3 SE +/- 0.0109, N = 3 5.4946 5.5020 5.5170 5.5224
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream a d c b 200 400 600 800 1000 SE +/- 0.46, N = 3 SE +/- 1.84, N = 3 807.17 806.32 806.23 804.40
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream a d c b 0.2792 0.5584 0.8376 1.1168 1.396 SE +/- 0.0007, N = 3 SE +/- 0.0029, N = 3 1.2364 1.2375 1.2381 1.2409
Neural Magic DeepSparse Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream a d b c 0.6606 1.3212 1.9818 2.6424 3.303 SE +/- 0.0343, N = 3 SE +/- 0.1622, N = 12 2.9358 2.9271 2.8607 2.5008
Neural Magic DeepSparse Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream a d b c 6K 12K 18K 24K 30K SE +/- 279.77, N = 3 SE +/- 1840.16, N = 12 24242.38 24331.06 24888.16 28968.95
Neural Magic DeepSparse Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream a b c d 5 10 15 20 25 SE +/- 0.08, N = 3 SE +/- 0.10, N = 3 20.83 20.81 20.73 20.46
Neural Magic DeepSparse Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream a b c d 11 22 33 44 55 SE +/- 0.18, N = 3 SE +/- 0.23, N = 3 47.97 48.03 48.22 48.84
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream a d c b 400 800 1200 1600 2000 SE +/- 1.97, N = 3 SE +/- 3.28, N = 3 1763.19 1758.31 1756.56 1756.17
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream a d c b 12 24 36 48 60 SE +/- 0.05, N = 3 SE +/- 0.11, N = 3 54.35 54.51 54.56 54.57
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream d b c a 50 100 150 200 250 SE +/- 0.62, N = 3 SE +/- 1.29, N = 3 209.19 208.72 208.63 207.67
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream d b c a 1.0828 2.1656 3.2484 4.3312 5.414 SE +/- 0.0142, N = 3 SE +/- 0.0294, N = 3 4.7773 4.7882 4.7905 4.8123
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream a d b c 200 400 600 800 1000 SE +/- 0.80, N = 3 SE +/- 1.69, N = 3 798.62 797.71 796.60 795.03
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream a d b c 30 60 90 120 150 SE +/- 0.13, N = 3 SE +/- 0.24, N = 3 119.91 120.00 120.18 120.40
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream d c b a 50 100 150 200 250 SE +/- 0.07, N = 3 SE +/- 0.31, N = 3 213.56 212.64 212.49 212.18
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream d c b a 1.0598 2.1196 3.1794 4.2392 5.299 SE +/- 0.0014, N = 3 SE +/- 0.0069, N = 3 4.6801 4.7002 4.7035 4.7104
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream a d b c 200 400 600 800 1000 SE +/- 2.26, N = 3 SE +/- 2.42, N = 3 1144.50 1144.01 1142.99 1141.24
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream a d b c 20 40 60 80 100 SE +/- 0.17, N = 3 SE +/- 0.17, N = 3 83.66 83.77 83.85 83.93
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream d a b c 50 100 150 200 250 SE +/- 0.36, N = 3 SE +/- 0.10, N = 3 225.56 225.47 225.45 225.44
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream d a b c 0.9974 1.9948 2.9922 3.9896 4.987 SE +/- 0.0072, N = 3 SE +/- 0.0020, N = 3 4.4307 4.4325 4.4329 4.4329
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream d a b c 50 100 150 200 250 SE +/- 0.46, N = 3 SE +/- 0.26, N = 3 249.78 249.72 249.40 249.09
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream a d b c 80 160 240 320 400 SE +/- 0.66, N = 3 SE +/- 0.66, N = 3 381.04 381.12 382.12 382.17
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream d c a b 15 30 45 60 75 SE +/- 0.15, N = 3 SE +/- 0.12, N = 3 65.00 64.92 64.89 64.86
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream d c a b 4 8 12 16 20 SE +/- 0.03, N = 3 SE +/- 0.03, N = 3 15.37 15.38 15.39 15.40
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream a c b d 600 1200 1800 2400 3000 SE +/- 1.45, N = 3 SE +/- 2.93, N = 3 2598.21 2595.48 2595.18 2592.85
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream a c b d 8 16 24 32 40 SE +/- 0.02, N = 3 SE +/- 0.04, N = 3 36.89 36.92 36.93 36.96
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream d a c b 15 30 45 60 75 SE +/- 0.09, N = 3 SE +/- 0.14, N = 3 68.77 68.58 68.56 68.40
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream d a c b 4 8 12 16 20 SE +/- 0.02, N = 3 SE +/- 0.03, N = 3 14.53 14.57 14.58 14.61
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream c a d b 30 60 90 120 150 SE +/- 0.72, N = 3 SE +/- 0.27, N = 3 133.81 132.95 132.91 132.80
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream c d a b 150 300 450 600 750 SE +/- 5.35, N = 3 SE +/- 1.53, N = 3 709.26 715.43 715.46 717.26
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream a b c d 11 22 33 44 55 SE +/- 0.01, N = 3 SE +/- 0.09, N = 3 48.53 48.42 48.38 48.38
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream a b c d 5 10 15 20 25 SE +/- 0.00, N = 3 SE +/- 0.04, N = 3 20.60 20.64 20.66 20.66
Primesieve Length: 1e12 OpenBenchmarking.org Seconds, Fewer Is Better Primesieve 12.1 Length: 1e12 d c a b 0.2619 0.5238 0.7857 1.0476 1.3095 SE +/- 0.008, N = 3 SE +/- 0.009, N = 3 1.142 1.156 1.164 1.164 1. (CXX) g++ options: -O3
Primesieve Length: 1e13 OpenBenchmarking.org Seconds, Fewer Is Better Primesieve 12.1 Length: 1e13 a d b c 3 6 9 12 15 SE +/- 0.02, N = 3 SE +/- 0.01, N = 3 11.79 11.84 11.85 11.87 1. (CXX) g++ options: -O3
SVT-AV1 Encoder Mode: Preset 4 - Input: Bosphorus 4K OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 2.0 Encoder Mode: Preset 4 - Input: Bosphorus 4K d b a c 2 4 6 8 10 SE +/- 0.017, N = 3 SE +/- 0.020, N = 3 8.634 8.575 8.557 8.484 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
SVT-AV1 Encoder Mode: Preset 8 - Input: Bosphorus 4K OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 2.0 Encoder Mode: Preset 8 - Input: Bosphorus 4K b c a d 20 40 60 80 100 SE +/- 0.83, N = 3 SE +/- 0.91, N = 3 91.29 90.44 89.28 86.59 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
SVT-AV1 Encoder Mode: Preset 12 - Input: Bosphorus 4K OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 2.0 Encoder Mode: Preset 12 - Input: Bosphorus 4K d b c a 40 80 120 160 200 SE +/- 1.72, N = 5 SE +/- 1.52, N = 15 163.50 163.18 160.56 149.07 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
SVT-AV1 Encoder Mode: Preset 13 - Input: Bosphorus 4K OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 2.0 Encoder Mode: Preset 13 - Input: Bosphorus 4K d c b a 40 80 120 160 200 SE +/- 1.62, N = 5 SE +/- 0.74, N = 3 162.38 162.02 161.65 159.22 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
SVT-AV1 Encoder Mode: Preset 4 - Input: Bosphorus 1080p OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 2.0 Encoder Mode: Preset 4 - Input: Bosphorus 1080p c a b d 6 12 18 24 30 SE +/- 0.24, N = 3 SE +/- 0.17, N = 3 23.47 22.96 22.90 22.71 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
SVT-AV1 Encoder Mode: Preset 8 - Input: Bosphorus 1080p OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 2.0 Encoder Mode: Preset 8 - Input: Bosphorus 1080p c d b a 40 80 120 160 200 SE +/- 2.01, N = 3 SE +/- 0.56, N = 3 184.13 183.29 182.53 176.67 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
SVT-AV1 Encoder Mode: Preset 12 - Input: Bosphorus 1080p OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 2.0 Encoder Mode: Preset 12 - Input: Bosphorus 1080p d a c b 120 240 360 480 600 SE +/- 5.33, N = 3 SE +/- 3.34, N = 3 577.44 572.35 568.21 566.99 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
SVT-AV1 Encoder Mode: Preset 13 - Input: Bosphorus 1080p OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 2.0 Encoder Mode: Preset 13 - Input: Bosphorus 1080p b d c a 130 260 390 520 650 SE +/- 4.06, N = 3 SE +/- 6.58, N = 3 606.38 598.04 588.45 564.41 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
Phoronix Test Suite v10.8.4