INTEL XEON PLATINUM 8592+ testing with a Quanta Cloud S6Q-MB-MPS (3B05.TEL4P1 BIOS) and ASPEED on Ubuntu 23.10 via the Phoronix Test Suite.
Compare your own system(s) to this result file with the
Phoronix Test Suite by running the command:
phoronix-test-suite benchmark 2312187-NE-NET50813920 net - Phoronix Test Suite net INTEL XEON PLATINUM 8592+ testing with a Quanta Cloud S6Q-MB-MPS (3B05.TEL4P1 BIOS) and ASPEED on Ubuntu 23.10 via the Phoronix Test Suite.
HTML result view exported from: https://openbenchmarking.org/result/2312187-NE-NET50813920&sor&grs .
net Processor Motherboard Chipset Memory Disk Graphics OS Kernel Desktop Display Server Compiler File-System Screen Resolution a b c INTEL XEON PLATINUM 8592+ @ 3.90GHz (64 Cores / 128 Threads) Quanta Cloud S6Q-MB-MPS (3B05.TEL4P1 BIOS) Intel Device 1bce 512GB 3201GB Micron_7450_MTFDKCC3T2TFS + 0GB Virtual HDisk0 + 0GB Virtual HDisk2 + 0GB Virtual HDisk1 + 0GB Virtual HDisk3 ASPEED Ubuntu 23.10 6.6.0-rc5-phx-patched (x86_64) GNOME Shell 45.0 X Server 1.21.1.7 GCC 13.2.0 ext4 1920x1200 OpenBenchmarking.org Kernel Details - Transparent Huge Pages: madvise Compiler Details - --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-13-XYspKM/gcc-13-13.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-13-XYspKM/gcc-13-13.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details - Scaling Governor: intel_pstate performance (EPP: performance) - CPU Microcode: 0x21000161 Java Details - a: OpenJDK Runtime Environment (build 11.0.20+8-post-Ubuntu-1ubuntu1) Python Details - Python 3.11.6 Security Details - gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected
net deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Stream svt-av1: Preset 8 - Bosphorus 1080p lczero: BLAS deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream svt-av1: Preset 4 - Bosphorus 1080p deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Stream deepsparse: ResNet-50, Baseline - Asynchronous Multi-Stream deepsparse: ResNet-50, Baseline - Asynchronous Multi-Stream deepsparse: ResNet-50, Baseline - Synchronous Single-Stream deepsparse: ResNet-50, Baseline - Synchronous Single-Stream svt-av1: Preset 8 - Bosphorus 4K xmrig: KawPow - 1M svt-av1: Preset 13 - Bosphorus 4K deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Stream xmrig: CryptoNight-Femto UPX2 - 1M deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream svt-av1: Preset 13 - Bosphorus 1080p deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Stream xmrig: Monero - 1M deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Stream xmrig: GhostRider - 1M deepsparse: CV Detection, YOLOv5s COCO - Synchronous Single-Stream deepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Stream svt-av1: Preset 12 - Bosphorus 1080p deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO - Synchronous Single-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream svt-av1: Preset 12 - Bosphorus 4K deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Stream xmrig: CryptoNight-Heavy - 1M deepsparse: BERT-Large, NLP Question Answering - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering - Synchronous Single-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream svt-av1: Preset 4 - Bosphorus 4K deepsparse: CV Detection, YOLOv5s COCO - Asynchronous Multi-Stream deepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO - Asynchronous Multi-Stream deepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Stream xmrig: Wownero - 1M deepsparse: BERT-Large, NLP Question Answering - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Stream lczero: Eigen a b c 28.4611 35.1023 139.811 67 5.3729 20.770 186.0257 96.1343 332.8063 34.2126 934.6524 3.4825 286.8882 75.139 39936.2 227.285 67.8694 14.7276 40425.9 34.2750 933.0074 632.459 29.0016 34.4749 210.9638 40329.6 4.7354 6862.9 164.9393 701.7915 49.8955 1.4200 545.910 641.1047 6.0577 28.8890 34.6092 290.0133 3.4452 228.874 71.9864 72.0462 442.2210 452.1951 70.7373 40394.9 31.2246 32.0197 442.3119 7.156 436.5226 5706.6964 73.2660 5.5953 166.6567 5.9970 43088.3 370.7287 86.3024 13.7504 2323.9799 34.0725 938.0661 796 28.8654 34.6174 141.977 67 5.3005 20.587 188.5376 95.6329 334.3153 34.4463 928.4462 3.4417 290.2826 75.737 40344.7 229.556 68.1760 14.6612 40080.4 34.2173 934.5224 636.909 28.9953 34.4826 210.8100 40153.2 4.7386 6857.8 164.8968 705.8268 50.1708 1.4123 543.725 637.6092 6.0589 28.8756 34.6250 288.6849 3.4609 229.514 71.6759 72.0875 441.4776 452.1641 70.7412 40282.2 31.1465 32.0999 443.3971 7.168 436.3286 5717.5185 73.2935 5.5844 166.5123 6.0023 42998.7 370.7761 86.2914 13.7462 2324.7362 34.0905 937.5775 805 29.6799 33.7246 141.873 68 5.3092 20.492 188.2324 96.8521 330.3352 34.0380 939.5363 3.4767 287.3469 74.926 40242.5 229.075 67.5802 14.7898 40331.1 34.0080 940.2198 637.337 29.2030 34.2377 209.6095 40072.2 4.7654 6820.8 163.9718 701.7061 50.1864 1.4205 546.877 637.4175 6.0922 29.0205 34.4526 289.8081 3.4473 228.518 71.9685 71.7827 443.1582 450.4918 71.0046 40286.6 31.1875 32.0585 442.7525 7.151 435.4932 5704.9530 73.4267 5.5965 166.3050 6.0096 43080.4 371.2957 86.1709 13.7597 2322.5803 34.0792 937.8630 817 OpenBenchmarking.org
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream a b c 7 14 21 28 35 SE +/- 0.12, N = 3 SE +/- 0.29, N = 3 SE +/- 0.34, N = 15 28.46 28.87 29.68
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream a b c 8 16 24 32 40 SE +/- 0.16, N = 3 SE +/- 0.35, N = 3 SE +/- 0.38, N = 15 35.10 34.62 33.72
SVT-AV1 Encoder Mode: Preset 8 - Input: Bosphorus 1080p OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 1.8 Encoder Mode: Preset 8 - Input: Bosphorus 1080p b c a 30 60 90 120 150 SE +/- 1.29, N = 7 SE +/- 1.86, N = 3 SE +/- 0.16, N = 3 141.98 141.87 139.81 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
LeelaChessZero Backend: BLAS OpenBenchmarking.org Nodes Per Second, More Is Better LeelaChessZero 0.30 Backend: BLAS c b a 15 30 45 60 75 SE +/- 0.58, N = 3 SE +/- 0.67, N = 3 SE +/- 0.67, N = 3 68 67 67 1. (CXX) g++ options: -flto -pthread
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream b c a 1.2089 2.4178 3.6267 4.8356 6.0445 SE +/- 0.0193, N = 3 SE +/- 0.0116, N = 3 SE +/- 0.0459, N = 3 5.3005 5.3092 5.3729
SVT-AV1 Encoder Mode: Preset 4 - Input: Bosphorus 1080p OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 1.8 Encoder Mode: Preset 4 - Input: Bosphorus 1080p a b c 5 10 15 20 25 SE +/- 0.08, N = 3 SE +/- 0.01, N = 3 SE +/- 0.25, N = 3 20.77 20.59 20.49 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream b c a 40 80 120 160 200 SE +/- 0.69, N = 3 SE +/- 0.41, N = 3 SE +/- 1.59, N = 3 188.54 188.23 186.03
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream c a b 20 40 60 80 100 SE +/- 0.08, N = 3 SE +/- 0.29, N = 3 SE +/- 0.27, N = 3 96.85 96.13 95.63
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream c a b 70 140 210 280 350 SE +/- 0.28, N = 3 SE +/- 1.00, N = 3 SE +/- 0.71, N = 3 330.34 332.81 334.32
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream c a b 8 16 24 32 40 SE +/- 0.03, N = 3 SE +/- 0.13, N = 3 SE +/- 0.33, N = 3 34.04 34.21 34.45
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream c a b 200 400 600 800 1000 SE +/- 0.83, N = 3 SE +/- 3.60, N = 3 SE +/- 8.90, N = 3 939.54 934.65 928.45
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream b c a 0.7836 1.5672 2.3508 3.1344 3.918 SE +/- 0.0077, N = 3 SE +/- 0.0147, N = 3 SE +/- 0.0119, N = 3 3.4417 3.4767 3.4825
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream b c a 60 120 180 240 300 SE +/- 0.65, N = 3 SE +/- 1.23, N = 3 SE +/- 0.98, N = 3 290.28 287.35 286.89
SVT-AV1 Encoder Mode: Preset 8 - Input: Bosphorus 4K OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 1.8 Encoder Mode: Preset 8 - Input: Bosphorus 4K b a c 20 40 60 80 100 SE +/- 0.27, N = 3 SE +/- 0.38, N = 3 SE +/- 0.15, N = 3 75.74 75.14 74.93 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
Xmrig Variant: KawPow - Hash Count: 1M OpenBenchmarking.org H/s, More Is Better Xmrig 6.21 Variant: KawPow - Hash Count: 1M b c a 9K 18K 27K 36K 45K SE +/- 97.76, N = 3 SE +/- 102.95, N = 3 SE +/- 185.86, N = 3 40344.7 40242.5 39936.2 1. (CXX) g++ options: -fexceptions -fno-rtti -maes -O3 -Ofast -static-libgcc -static-libstdc++ -rdynamic -lssl -lcrypto -luv -lpthread -lrt -ldl -lhwloc
SVT-AV1 Encoder Mode: Preset 13 - Input: Bosphorus 4K OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 1.8 Encoder Mode: Preset 13 - Input: Bosphorus 4K b c a 50 100 150 200 250 SE +/- 0.75, N = 3 SE +/- 0.19, N = 3 SE +/- 2.23, N = 3 229.56 229.08 227.29 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream b a c 15 30 45 60 75 SE +/- 0.12, N = 3 SE +/- 0.27, N = 3 SE +/- 0.11, N = 3 68.18 67.87 67.58
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream b a c 4 8 12 16 20 SE +/- 0.03, N = 3 SE +/- 0.06, N = 3 SE +/- 0.02, N = 3 14.66 14.73 14.79
Xmrig Variant: CryptoNight-Femto UPX2 - Hash Count: 1M OpenBenchmarking.org H/s, More Is Better Xmrig 6.21 Variant: CryptoNight-Femto UPX2 - Hash Count: 1M a c b 9K 18K 27K 36K 45K SE +/- 55.21, N = 3 SE +/- 88.91, N = 3 SE +/- 72.11, N = 3 40425.9 40331.1 40080.4 1. (CXX) g++ options: -fexceptions -fno-rtti -maes -O3 -Ofast -static-libgcc -static-libstdc++ -rdynamic -lssl -lcrypto -luv -lpthread -lrt -ldl -lhwloc
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream c b a 8 16 24 32 40 SE +/- 0.01, N = 3 SE +/- 0.17, N = 3 SE +/- 0.14, N = 3 34.01 34.22 34.28
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream c b a 200 400 600 800 1000 SE +/- 0.19, N = 3 SE +/- 4.66, N = 3 SE +/- 3.85, N = 3 940.22 934.52 933.01
SVT-AV1 Encoder Mode: Preset 13 - Input: Bosphorus 1080p OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 1.8 Encoder Mode: Preset 13 - Input: Bosphorus 1080p c b a 140 280 420 560 700 SE +/- 4.46, N = 3 SE +/- 4.83, N = 3 SE +/- 3.46, N = 3 637.34 636.91 632.46 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream b a c 7 14 21 28 35 SE +/- 0.05, N = 3 SE +/- 0.02, N = 3 SE +/- 0.06, N = 3 29.00 29.00 29.20
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream b a c 8 16 24 32 40 SE +/- 0.05, N = 3 SE +/- 0.02, N = 3 SE +/- 0.07, N = 3 34.48 34.47 34.24
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream a b c 50 100 150 200 250 SE +/- 1.74, N = 3 SE +/- 0.83, N = 3 SE +/- 0.49, N = 3 210.96 210.81 209.61
Xmrig Variant: Monero - Hash Count: 1M OpenBenchmarking.org H/s, More Is Better Xmrig 6.21 Variant: Monero - Hash Count: 1M a b c 9K 18K 27K 36K 45K SE +/- 106.27, N = 3 SE +/- 234.97, N = 3 SE +/- 234.22, N = 3 40329.6 40153.2 40072.2 1. (CXX) g++ options: -fexceptions -fno-rtti -maes -O3 -Ofast -static-libgcc -static-libstdc++ -rdynamic -lssl -lcrypto -luv -lpthread -lrt -ldl -lhwloc
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream a b c 1.0722 2.1444 3.2166 4.2888 5.361 SE +/- 0.0394, N = 3 SE +/- 0.0188, N = 3 SE +/- 0.0109, N = 3 4.7354 4.7386 4.7654
Xmrig Variant: GhostRider - Hash Count: 1M OpenBenchmarking.org H/s, More Is Better Xmrig 6.21 Variant: GhostRider - Hash Count: 1M a b c 1500 3000 4500 6000 7500 SE +/- 5.37, N = 3 SE +/- 29.78, N = 3 SE +/- 18.86, N = 3 6862.9 6857.8 6820.8 1. (CXX) g++ options: -fexceptions -fno-rtti -maes -O3 -Ofast -static-libgcc -static-libstdc++ -rdynamic -lssl -lcrypto -luv -lpthread -lrt -ldl -lhwloc
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream a b c 40 80 120 160 200 SE +/- 0.67, N = 3 SE +/- 0.12, N = 3 SE +/- 0.23, N = 3 164.94 164.90 163.97
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream b a c 150 300 450 600 750 SE +/- 2.96, N = 3 SE +/- 2.07, N = 3 SE +/- 0.26, N = 3 705.83 701.79 701.71
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream a b c 11 22 33 44 55 SE +/- 0.03, N = 3 SE +/- 0.15, N = 3 SE +/- 0.17, N = 3 49.90 50.17 50.19
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream b a c 0.3196 0.6392 0.9588 1.2784 1.598 SE +/- 0.0057, N = 3 SE +/- 0.0038, N = 3 SE +/- 0.0007, N = 3 1.4123 1.4200 1.4205
SVT-AV1 Encoder Mode: Preset 12 - Input: Bosphorus 1080p OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 1.8 Encoder Mode: Preset 12 - Input: Bosphorus 1080p c a b 120 240 360 480 600 SE +/- 4.85, N = 3 SE +/- 4.95, N = 3 SE +/- 3.52, N = 3 546.88 545.91 543.73 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream a b c 140 280 420 560 700 SE +/- 0.45, N = 3 SE +/- 1.94, N = 3 SE +/- 2.11, N = 3 641.10 637.61 637.42
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream a b c 2 4 6 8 10 SE +/- 0.0247, N = 3 SE +/- 0.0043, N = 3 SE +/- 0.0081, N = 3 6.0577 6.0589 6.0922
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream b a c 7 14 21 28 35 SE +/- 0.02, N = 3 SE +/- 0.03, N = 3 SE +/- 0.03, N = 3 28.88 28.89 29.02
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream b a c 8 16 24 32 40 SE +/- 0.02, N = 3 SE +/- 0.04, N = 3 SE +/- 0.03, N = 3 34.63 34.61 34.45
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream a c b 60 120 180 240 300 SE +/- 1.63, N = 3 SE +/- 0.98, N = 3 SE +/- 0.99, N = 3 290.01 289.81 288.68
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream a c b 0.7787 1.5574 2.3361 3.1148 3.8935 SE +/- 0.0193, N = 3 SE +/- 0.0117, N = 3 SE +/- 0.0119, N = 3 3.4452 3.4473 3.4609
SVT-AV1 Encoder Mode: Preset 12 - Input: Bosphorus 4K OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 1.8 Encoder Mode: Preset 12 - Input: Bosphorus 4K b a c 50 100 150 200 250 SE +/- 1.07, N = 3 SE +/- 0.42, N = 3 SE +/- 1.46, N = 3 229.51 228.87 228.52 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream a c b 16 32 48 64 80 SE +/- 0.16, N = 3 SE +/- 0.20, N = 3 SE +/- 0.17, N = 3 71.99 71.97 71.68
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream b a c 16 32 48 64 80 SE +/- 0.04, N = 3 SE +/- 0.17, N = 3 SE +/- 0.07, N = 3 72.09 72.05 71.78
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream b a c 100 200 300 400 500 SE +/- 0.59, N = 3 SE +/- 0.73, N = 3 SE +/- 0.25, N = 3 441.48 442.22 443.16
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c 100 200 300 400 500 SE +/- 0.30, N = 3 SE +/- 0.02, N = 3 SE +/- 0.54, N = 3 452.20 452.16 450.49
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c 16 32 48 64 80 SE +/- 0.05, N = 3 SE +/- 0.00, N = 3 SE +/- 0.09, N = 3 70.74 70.74 71.00
Xmrig Variant: CryptoNight-Heavy - Hash Count: 1M OpenBenchmarking.org H/s, More Is Better Xmrig 6.21 Variant: CryptoNight-Heavy - Hash Count: 1M a c b 9K 18K 27K 36K 45K SE +/- 42.10, N = 3 SE +/- 169.87, N = 3 SE +/- 64.66, N = 3 40394.9 40286.6 40282.2 1. (CXX) g++ options: -fexceptions -fno-rtti -maes -O3 -Ofast -static-libgcc -static-libstdc++ -rdynamic -lssl -lcrypto -luv -lpthread -lrt -ldl -lhwloc
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream a c b 7 14 21 28 35 SE +/- 0.02, N = 3 SE +/- 0.08, N = 3 SE +/- 0.03, N = 3 31.22 31.19 31.15
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream a c b 7 14 21 28 35 SE +/- 0.02, N = 3 SE +/- 0.08, N = 3 SE +/- 0.03, N = 3 32.02 32.06 32.10
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream a c b 100 200 300 400 500 SE +/- 0.57, N = 3 SE +/- 0.77, N = 3 SE +/- 0.20, N = 3 442.31 442.75 443.40
SVT-AV1 Encoder Mode: Preset 4 - Input: Bosphorus 4K OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 1.8 Encoder Mode: Preset 4 - Input: Bosphorus 4K b a c 2 4 6 8 10 SE +/- 0.025, N = 3 SE +/- 0.022, N = 3 SE +/- 0.071, N = 3 7.168 7.156 7.151 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream a b c 90 180 270 360 450 SE +/- 0.33, N = 3 SE +/- 0.34, N = 3 SE +/- 0.19, N = 3 436.52 436.33 435.49
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream b a c 1200 2400 3600 4800 6000 SE +/- 10.52, N = 3 SE +/- 15.58, N = 3 SE +/- 16.89, N = 3 5717.52 5706.70 5704.95
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream a b c 16 32 48 64 80 SE +/- 0.05, N = 3 SE +/- 0.05, N = 3 SE +/- 0.03, N = 3 73.27 73.29 73.43
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream b a c 1.2592 2.5184 3.7776 5.0368 6.296 SE +/- 0.0102, N = 3 SE +/- 0.0153, N = 3 SE +/- 0.0160, N = 3 5.5844 5.5953 5.5965
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream a b c 40 80 120 160 200 SE +/- 0.33, N = 3 SE +/- 0.25, N = 3 SE +/- 0.19, N = 3 166.66 166.51 166.31
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream a b c 2 4 6 8 10 SE +/- 0.0119, N = 3 SE +/- 0.0087, N = 3 SE +/- 0.0069, N = 3 5.9970 6.0023 6.0096
Xmrig Variant: Wownero - Hash Count: 1M OpenBenchmarking.org H/s, More Is Better Xmrig 6.21 Variant: Wownero - Hash Count: 1M a c b 9K 18K 27K 36K 45K SE +/- 76.66, N = 3 SE +/- 104.95, N = 3 SE +/- 81.91, N = 3 43088.3 43080.4 42998.7 1. (CXX) g++ options: -fexceptions -fno-rtti -maes -O3 -Ofast -static-libgcc -static-libstdc++ -rdynamic -lssl -lcrypto -luv -lpthread -lrt -ldl -lhwloc
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream a b c 80 160 240 320 400 SE +/- 0.59, N = 3 SE +/- 0.36, N = 3 SE +/- 0.22, N = 3 370.73 370.78 371.30
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream a b c 20 40 60 80 100 SE +/- 0.14, N = 3 SE +/- 0.09, N = 3 SE +/- 0.05, N = 3 86.30 86.29 86.17
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream b a c 4 8 12 16 20 SE +/- 0.02, N = 3 SE +/- 0.01, N = 3 SE +/- 0.00, N = 3 13.75 13.75 13.76
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream b a c 500 1000 1500 2000 2500 SE +/- 2.92, N = 3 SE +/- 2.42, N = 3 SE +/- 0.54, N = 3 2324.74 2323.98 2322.58
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream a c b 8 16 24 32 40 SE +/- 0.01, N = 3 SE +/- 0.00, N = 3 SE +/- 0.00, N = 3 34.07 34.08 34.09
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream a c b 200 400 600 800 1000 SE +/- 0.34, N = 3 SE +/- 0.07, N = 3 SE +/- 0.09, N = 3 938.07 937.86 937.58
LeelaChessZero Backend: Eigen OpenBenchmarking.org Nodes Per Second, More Is Better LeelaChessZero 0.30 Backend: Eigen c b a 200 400 600 800 1000 SE +/- 32.94, N = 6 SE +/- 11.08, N = 9 SE +/- 21.82, N = 9 817 805 796 1. (CXX) g++ options: -flto -pthread
Phoronix Test Suite v10.8.4