2 x AMD EPYC 9684X 96-Core testing with a AMD Titanite_4G (RTI1007B BIOS) and ASPEED on Ubuntu 23.10 via the Phoronix Test Suite.
Compare your own system(s) to this result file with the
Phoronix Test Suite by running the command:
phoronix-test-suite benchmark 2403164-NE-DDFG2505160 ddfg - Phoronix Test Suite ddfg 2 x AMD EPYC 9684X 96-Core testing with a AMD Titanite_4G (RTI1007B BIOS) and ASPEED on Ubuntu 23.10 via the Phoronix Test Suite.
HTML result view exported from: https://openbenchmarking.org/result/2403164-NE-DDFG2505160&grs&rdt .
ddfg Processor Motherboard Chipset Memory Disk Graphics Network OS Kernel Compiler File-System Screen Resolution a b c d 2 x AMD EPYC 9684X 96-Core @ 2.55GHz (192 Cores / 384 Threads) AMD Titanite_4G (RTI1007B BIOS) AMD Device 14a4 1520GB 3201GB Micron_7450_MTFDKCB3T2TFS + 257GB Flash Drive ASPEED Broadcom NetXtreme BCM5720 PCIe Ubuntu 23.10 6.5.0-25-generic (x86_64) GCC 13.2.0 ext4 640x480 OpenBenchmarking.org Kernel Details - Transparent Huge Pages: madvise Compiler Details - --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-13-XYspKM/gcc-13-13.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-13-XYspKM/gcc-13-13.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details - Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa10113e Python Details - Python 3.11.6 Security Details - gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_rstack_overflow: Mitigation of Safe RET + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected
ddfg svt-av1: Preset 12 - Bosphorus 4K svt-av1: Preset 13 - Bosphorus 1080p svt-av1: Preset 8 - Bosphorus 4K svt-av1: Preset 8 - Bosphorus 1080p svt-av1: Preset 4 - Bosphorus 1080p svt-av1: Preset 13 - Bosphorus 4K primesieve: 1e12 svt-av1: Preset 12 - Bosphorus 1080p deepsparse: Llama2 Chat 7b Quantized - Synchronous Single-Stream deepsparse: Llama2 Chat 7b Quantized - Synchronous Single-Stream svt-av1: Preset 4 - Bosphorus 4K deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Stream deepsparse: ResNet-50, Baseline - Synchronous Single-Stream deepsparse: ResNet-50, Baseline - Synchronous Single-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream primesieve: 1e13 deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Stream deepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Stream deepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream deepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Stream deepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Stream deepsparse: ResNet-50, Baseline - Asynchronous Multi-Stream deepsparse: ResNet-50, Baseline - Asynchronous Multi-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream deepsparse: Llama2 Chat 7b Quantized - Asynchronous Multi-Stream deepsparse: Llama2 Chat 7b Quantized - Asynchronous Multi-Stream a b c d 149.069 564.413 89.277 176.674 22.964 159.218 1.164 572.349 20.8303 47.9735 8.557 5.218 191.5137 206.8154 4.8321 715.4595 132.9467 4.8123 207.6733 11.789 212.1812 4.7104 5566.3853 17.2212 713.5443 14.5734 68.5822 17421.4665 5.4946 798.6248 119.9052 1763.1949 54.3534 1.2364 807.1699 48.516 20.6047 83.6574 20.5991 48.5284 381.0376 1144.5034 249.7234 1762.5286 54.3739 64.8907 15.3899 2598.2083 36.8891 132.9523 225.4674 4.4325 24242.3848 2.9358 163.175 606.376 91.288 182.531 22.898 161.653 1.164 566.993 20.8052 48.0337 8.575 5.2109 191.7714 209.7711 4.7642 717.2565 132.7968 4.7882 208.7235 11.848 212.4945 4.7035 5538.9806 17.3043 717.4900 14.6134 68.3972 17350.4640 5.5170 796.5966 120.1787 1756.1742 54.5660 1.2409 804.4025 48.3555 20.6727 83.8482 20.6441 48.4228 382.1237 1142.9944 249.4018 1758.8057 54.4869 64.8586 15.3982 2595.1763 36.9324 132.9817 225.4496 4.4329 24888.1563 2.8607 160.560 588.450 90.441 184.132 23.474 162.020 1.156 568.209 20.7259 48.2177 8.484 5.1985 192.2335 208.7406 4.7875 709.2588 133.8061 4.7905 208.6327 11.871 212.6390 4.7002 5534.2604 17.3188 715.8770 14.5791 68.5565 17333.3501 5.5224 795.0277 120.3976 1756.5604 54.5610 1.2381 806.2338 48.5178 20.6037 83.9259 20.6607 48.3841 382.1742 1141.2417 249.0943 1759.9562 54.4588 64.9222 15.3844 2595.4788 36.9216 132.8792 225.4415 4.4329 28968.9536 2.5008 163.5 598.04 86.593 183.285 22.713 162.377 1.142 577.444 20.4622 48.8364 8.634 5.1383 194.4729 208.9767 4.7823 715.4294 132.9148 4.7773 209.1911 11.841 213.5555 4.6801 5554.0345 17.257 714.456 14.5348 68.7657 17400.5797 5.502 797.7062 119.9993 1758.3058 54.505 1.2375 806.3219 48.5095 20.6072 83.7657 20.6629 48.379 381.1234 1144.0054 249.7754 1763.4247 54.3609 64.997 15.366 2592.8516 36.9564 133.012 225.5614 4.4307 24331.0626 2.9271 OpenBenchmarking.org
SVT-AV1 Encoder Mode: Preset 12 - Input: Bosphorus 4K OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 2.0 Encoder Mode: Preset 12 - Input: Bosphorus 4K a b c d 40 80 120 160 200 SE +/- 1.72, N = 5 SE +/- 1.52, N = 15 149.07 163.18 160.56 163.50 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
SVT-AV1 Encoder Mode: Preset 13 - Input: Bosphorus 1080p OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 2.0 Encoder Mode: Preset 13 - Input: Bosphorus 1080p a b c d 130 260 390 520 650 SE +/- 4.06, N = 3 SE +/- 6.58, N = 3 564.41 606.38 588.45 598.04 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
SVT-AV1 Encoder Mode: Preset 8 - Input: Bosphorus 4K OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 2.0 Encoder Mode: Preset 8 - Input: Bosphorus 4K a b c d 20 40 60 80 100 SE +/- 0.83, N = 3 SE +/- 0.91, N = 3 89.28 91.29 90.44 86.59 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
SVT-AV1 Encoder Mode: Preset 8 - Input: Bosphorus 1080p OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 2.0 Encoder Mode: Preset 8 - Input: Bosphorus 1080p a b c d 40 80 120 160 200 SE +/- 0.56, N = 3 SE +/- 2.01, N = 3 176.67 182.53 184.13 183.29 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
SVT-AV1 Encoder Mode: Preset 4 - Input: Bosphorus 1080p OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 2.0 Encoder Mode: Preset 4 - Input: Bosphorus 1080p a b c d 6 12 18 24 30 SE +/- 0.17, N = 3 SE +/- 0.24, N = 3 22.96 22.90 23.47 22.71 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
SVT-AV1 Encoder Mode: Preset 13 - Input: Bosphorus 4K OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 2.0 Encoder Mode: Preset 13 - Input: Bosphorus 4K a b c d 40 80 120 160 200 SE +/- 0.74, N = 3 SE +/- 1.62, N = 5 159.22 161.65 162.02 162.38 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
Primesieve Length: 1e12 OpenBenchmarking.org Seconds, Fewer Is Better Primesieve 12.1 Length: 1e12 a b c d 0.2619 0.5238 0.7857 1.0476 1.3095 SE +/- 0.009, N = 3 SE +/- 0.008, N = 3 1.164 1.164 1.156 1.142 1. (CXX) g++ options: -O3
SVT-AV1 Encoder Mode: Preset 12 - Input: Bosphorus 1080p OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 2.0 Encoder Mode: Preset 12 - Input: Bosphorus 1080p a b c d 120 240 360 480 600 SE +/- 3.34, N = 3 SE +/- 5.33, N = 3 572.35 566.99 568.21 577.44 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
Neural Magic DeepSparse Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream a b c d 5 10 15 20 25 SE +/- 0.08, N = 3 SE +/- 0.10, N = 3 20.83 20.81 20.73 20.46
Neural Magic DeepSparse Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream a b c d 11 22 33 44 55 SE +/- 0.18, N = 3 SE +/- 0.23, N = 3 47.97 48.03 48.22 48.84
SVT-AV1 Encoder Mode: Preset 4 - Input: Bosphorus 4K OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 2.0 Encoder Mode: Preset 4 - Input: Bosphorus 4K a b c d 2 4 6 8 10 SE +/- 0.017, N = 3 SE +/- 0.020, N = 3 8.557 8.575 8.484 8.634 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 1.1741 2.3482 3.5223 4.6964 5.8705 SE +/- 0.0177, N = 3 SE +/- 0.0167, N = 3 5.2180 5.2109 5.1985 5.1383
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 40 80 120 160 200 SE +/- 0.65, N = 3 SE +/- 0.62, N = 3 191.51 191.77 192.23 194.47
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream a b c d 50 100 150 200 250 SE +/- 0.44, N = 3 SE +/- 0.35, N = 3 206.82 209.77 208.74 208.98
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream a b c d 1.0872 2.1744 3.2616 4.3488 5.436 SE +/- 0.0100, N = 3 SE +/- 0.0081, N = 3 4.8321 4.7642 4.7875 4.7823
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream a b c d 150 300 450 600 750 SE +/- 1.53, N = 3 SE +/- 5.35, N = 3 715.46 717.26 709.26 715.43
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream a b c d 30 60 90 120 150 SE +/- 0.27, N = 3 SE +/- 0.72, N = 3 132.95 132.80 133.81 132.91
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream a b c d 1.0828 2.1656 3.2484 4.3312 5.414 SE +/- 0.0142, N = 3 SE +/- 0.0294, N = 3 4.8123 4.7882 4.7905 4.7773
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream a b c d 50 100 150 200 250 SE +/- 0.62, N = 3 SE +/- 1.29, N = 3 207.67 208.72 208.63 209.19
Primesieve Length: 1e13 OpenBenchmarking.org Seconds, Fewer Is Better Primesieve 12.1 Length: 1e13 a b c d 3 6 9 12 15 SE +/- 0.02, N = 3 SE +/- 0.01, N = 3 11.79 11.85 11.87 11.84 1. (CXX) g++ options: -O3
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 50 100 150 200 250 SE +/- 0.31, N = 3 SE +/- 0.07, N = 3 212.18 212.49 212.64 213.56
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 1.0598 2.1196 3.1794 4.2392 5.299 SE +/- 0.0069, N = 3 SE +/- 0.0014, N = 3 4.7104 4.7035 4.7002 4.6801
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 1200 2400 3600 4800 6000 SE +/- 6.35, N = 3 SE +/- 7.14, N = 3 5566.39 5538.98 5534.26 5554.03
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 4 8 12 16 20 SE +/- 0.02, N = 3 SE +/- 0.02, N = 3 17.22 17.30 17.32 17.26
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream a b c d 150 300 450 600 750 SE +/- 1.03, N = 3 SE +/- 3.41, N = 3 713.54 717.49 715.88 714.46
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 4 8 12 16 20 SE +/- 0.03, N = 3 SE +/- 0.02, N = 3 14.57 14.61 14.58 14.53
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 15 30 45 60 75 SE +/- 0.14, N = 3 SE +/- 0.09, N = 3 68.58 68.40 68.56 68.77
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 4K 8K 12K 16K 20K SE +/- 17.22, N = 3 SE +/- 34.07, N = 3 17421.47 17350.46 17333.35 17400.58
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 1.2425 2.485 3.7275 4.97 6.2125 SE +/- 0.0052, N = 3 SE +/- 0.0109, N = 3 5.4946 5.5170 5.5224 5.5020
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 200 400 600 800 1000 SE +/- 0.80, N = 3 SE +/- 1.69, N = 3 798.62 796.60 795.03 797.71
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 30 60 90 120 150 SE +/- 0.13, N = 3 SE +/- 0.24, N = 3 119.91 120.18 120.40 120.00
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream a b c d 400 800 1200 1600 2000 SE +/- 3.28, N = 3 SE +/- 1.97, N = 3 1763.19 1756.17 1756.56 1758.31
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream a b c d 12 24 36 48 60 SE +/- 0.11, N = 3 SE +/- 0.05, N = 3 54.35 54.57 54.56 54.51
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 0.2792 0.5584 0.8376 1.1168 1.396 SE +/- 0.0029, N = 3 SE +/- 0.0007, N = 3 1.2364 1.2409 1.2381 1.2375
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 200 400 600 800 1000 SE +/- 1.84, N = 3 SE +/- 0.46, N = 3 807.17 804.40 806.23 806.32
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream a b c d 11 22 33 44 55 SE +/- 0.03, N = 3 SE +/- 0.04, N = 3 48.52 48.36 48.52 48.51
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream a b c d 5 10 15 20 25 SE +/- 0.01, N = 3 SE +/- 0.02, N = 3 20.60 20.67 20.60 20.61
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream a b c d 20 40 60 80 100 SE +/- 0.17, N = 3 SE +/- 0.17, N = 3 83.66 83.85 83.93 83.77
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream a b c d 5 10 15 20 25 SE +/- 0.00, N = 3 SE +/- 0.04, N = 3 20.60 20.64 20.66 20.66
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream a b c d 11 22 33 44 55 SE +/- 0.01, N = 3 SE +/- 0.09, N = 3 48.53 48.42 48.38 48.38
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream a b c d 80 160 240 320 400 SE +/- 0.66, N = 3 SE +/- 0.66, N = 3 381.04 382.12 382.17 381.12
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream a b c d 200 400 600 800 1000 SE +/- 2.26, N = 3 SE +/- 2.42, N = 3 1144.50 1142.99 1141.24 1144.01
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream a b c d 50 100 150 200 250 SE +/- 0.46, N = 3 SE +/- 0.26, N = 3 249.72 249.40 249.09 249.78
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream a b c d 400 800 1200 1600 2000 SE +/- 2.58, N = 3 SE +/- 2.77, N = 3 1762.53 1758.81 1759.96 1763.42
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream a b c d 12 24 36 48 60 SE +/- 0.09, N = 3 SE +/- 0.08, N = 3 54.37 54.49 54.46 54.36
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream a b c d 15 30 45 60 75 SE +/- 0.12, N = 3 SE +/- 0.15, N = 3 64.89 64.86 64.92 65.00
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream a b c d 4 8 12 16 20 SE +/- 0.03, N = 3 SE +/- 0.03, N = 3 15.39 15.40 15.38 15.37
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 600 1200 1800 2400 3000 SE +/- 2.93, N = 3 SE +/- 1.45, N = 3 2598.21 2595.18 2595.48 2592.85
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 8 16 24 32 40 SE +/- 0.04, N = 3 SE +/- 0.02, N = 3 36.89 36.93 36.92 36.96
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream a b c d 30 60 90 120 150 SE +/- 0.21, N = 3 SE +/- 0.23, N = 3 132.95 132.98 132.88 133.01
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream a b c d 50 100 150 200 250 SE +/- 0.36, N = 3 SE +/- 0.10, N = 3 225.47 225.45 225.44 225.56
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream a b c d 0.9974 1.9948 2.9922 3.9896 4.987 SE +/- 0.0072, N = 3 SE +/- 0.0020, N = 3 4.4325 4.4329 4.4329 4.4307
Neural Magic DeepSparse Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream a b c d 6K 12K 18K 24K 30K SE +/- 279.77, N = 3 SE +/- 1840.16, N = 12 24242.38 24888.16 28968.95 24331.06
Neural Magic DeepSparse Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream a b c d 0.6606 1.3212 1.9818 2.6424 3.303 SE +/- 0.0343, N = 3 SE +/- 0.1622, N = 12 2.9358 2.8607 2.5008 2.9271
Phoronix Test Suite v10.8.4