a

Benchmarks for a future article.

HTML result view exported from: https://openbenchmarking.org/result/2312143-NE-A8154652071&grw.

aProcessorMotherboardChipsetMemoryDiskGraphicsNetworkOSKernelCompilerFile-SystemScreen Resolutionabcd2 x INTEL XEON PLATINUM 8592+ @ 3.90GHz (128 Cores / 256 Threads)Quanta Cloud S6Q-MB-MPS (3B05.TEL4P1 BIOS)Intel Device 1bce1008GB3201GB Micron_7450_MTFDKCB3T2TFSASPEED2 x Intel X710 for 10GBASE-TUbuntu 23.106.5.0-13-generic (x86_64)GCC 13.2.0ext41920x1080OpenBenchmarking.orgKernel Details- Transparent Huge Pages: madviseProcessor Details- Scaling Governor: intel_pstate powersave (EPP: balance_performance) - CPU Microcode: 0x21000161 Python Details- Python 3.11.6Security Details- gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected Compiler Details- d: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-13-XYspKM/gcc-13-13.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-13-XYspKM/gcc-13-13.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v

awrf: conus 2.5kmdeepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Streamdeepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Streamdeepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Streamdeepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Streamdeepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Streamdeepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Streamdeepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Streamdeepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Streamdeepsparse: ResNet-50, Baseline - Asynchronous Multi-Streamdeepsparse: ResNet-50, Baseline - Asynchronous Multi-Streamdeepsparse: ResNet-50, Baseline - Synchronous Single-Streamdeepsparse: ResNet-50, Baseline - Synchronous Single-Streamdeepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Streamdeepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Streamdeepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Streamdeepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Streamdeepsparse: CV Detection, YOLOv5s COCO - Asynchronous Multi-Streamdeepsparse: CV Detection, YOLOv5s COCO - Asynchronous Multi-Streamdeepsparse: CV Detection, YOLOv5s COCO - Synchronous Single-Streamdeepsparse: CV Detection, YOLOv5s COCO - Synchronous Single-Streamdeepsparse: BERT-Large, NLP Question Answering - Asynchronous Multi-Streamdeepsparse: BERT-Large, NLP Question Answering - Asynchronous Multi-Streamdeepsparse: BERT-Large, NLP Question Answering - Synchronous Single-Streamdeepsparse: BERT-Large, NLP Question Answering - Synchronous Single-Streamdeepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Streamdeepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Streamdeepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Streamdeepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Streamdeepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Streamdeepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Streamdeepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Streamdeepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Streamdeepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Streamdeepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Streamdeepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Streamdeepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Streamdeepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Streamdeepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Streamdeepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Streamdeepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Streamdeepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Streamdeepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Streamdeepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Streamdeepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Streamdeepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Streamdeepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Streamdeepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Streamdeepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Streamnwchem: C240 Buckyballabcd5566.729133.5159475.696431.194532.06134554.979614.0362238.65234.20831829.964434.9169338.95622.949211228.66255.68521030.31810.9681828.038477.1758190.84205.2382156.0563407.277729.942133.40221832.792734.8681339.04162.9484854.052574.8177194.24765.14851231.204951.9301199.69305.0073183.4966347.663734.962028.58761880.768833.980896.430010.3634133.6023475.126531.695431.582117445600.976133.7189474.185231.233632.03684563.509714.0081237.04604.22951824.375935.0063337.67992.960211169.85175.71401027.51170.9706828.653277.0924190.03995.2617155.7222408.626330.389132.89891817.993935.1423338.85832.9500852.945574.9435193.59725.16591231.293951.9273199.33805.0196180.3157353.688935.106528.47011875.903534.081596.379010.3695133.8239474.368731.533231.73031730.75583.112133.4843475.680431.630031.65014563.427814.0090230.76404.34361827.978034.9683338.55362.953011163.65935.71651043.28040.9556828.446277.1235189.81755.2651155.7677408.969930.294432.99841834.477134.8217338.21922.9555854.252174.7945193.08725.18021231.273651.9192199.33235.0179177.1512359.891935.005728.55211879.838833.993996.547410.3507133.6004475.349731.720231.56181757.35617.197133.0079477.610832.893630.42224556.023314.0315240.23364.16041837.497734.7567338.14662.957311221.13575.69081032.24100.9667825.084177.4573190.13765.2589155.6020408.163829.523133.86351819.686135.1013339.36712.9455850.892675.0848193.81865.16141226.796052.0706196.78745.0808181.2029352.223534.998728.55861865.019334.250895.698110.4501133.5969475.446433.054030.28911748OpenBenchmarking.org

WRF

Input: conus 2.5km

OpenBenchmarking.orgSeconds, Fewer Is BetterWRF 4.2.2Input: conus 2.5kmabcd120024003600480060005566.735600.985583.115617.201. (F9X) gfortran options: -O2 -ftree-vectorize -funroll-loops -ffree-form -fconvert=big-endian -frecord-marker=4 -fallow-invalid-boz -lesmf_time -lwrfio_nf -lnetcdff -lnetcdf -lmpi_usempif08 -lmpi_mpifh -lmpi -lopen-rte -lopen-pal -lhwloc -levent_core -levent_pthreads -lm -lz

Neural Magic DeepSparse

Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Streamabcd306090120150SE +/- 0.15, N = 3SE +/- 0.13, N = 3SE +/- 0.16, N = 3SE +/- 0.11, N = 3133.52133.72133.48133.01

Neural Magic DeepSparse

Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Streamabcd100200300400500SE +/- 0.61, N = 3SE +/- 0.47, N = 3SE +/- 0.52, N = 3SE +/- 0.44, N = 3475.70474.19475.68477.61

Neural Magic DeepSparse

Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Streamabcd816243240SE +/- 0.38, N = 4SE +/- 0.25, N = 15SE +/- 0.32, N = 15SE +/- 0.26, N = 1531.1931.2331.6332.89

Neural Magic DeepSparse

Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Streamabcd714212835SE +/- 0.40, N = 4SE +/- 0.27, N = 15SE +/- 0.32, N = 15SE +/- 0.25, N = 1532.0632.0431.6530.42

Neural Magic DeepSparse

Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Streamabcd10002000300040005000SE +/- 49.40, N = 5SE +/- 40.39, N = 7SE +/- 43.60, N = 6SE +/- 42.83, N = 64554.984563.514563.434556.02

Neural Magic DeepSparse

Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Streamabcd48121620SE +/- 0.16, N = 5SE +/- 0.13, N = 7SE +/- 0.14, N = 6SE +/- 0.14, N = 614.0414.0114.0114.03

Neural Magic DeepSparse

Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Streamabcd50100150200250SE +/- 4.61, N = 15SE +/- 3.82, N = 15SE +/- 3.66, N = 15SE +/- 3.16, N = 3238.65237.05230.76240.23

Neural Magic DeepSparse

Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Streamabcd0.97731.95462.93193.90924.8865SE +/- 0.0821, N = 15SE +/- 0.0666, N = 15SE +/- 0.0661, N = 15SE +/- 0.0540, N = 34.20834.22954.34364.1604

Neural Magic DeepSparse

Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Streamabcd400800120016002000SE +/- 4.85, N = 3SE +/- 13.19, N = 3SE +/- 17.94, N = 3SE +/- 14.66, N = 31829.961824.381827.981837.50

Neural Magic DeepSparse

Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Streamabcd816243240SE +/- 0.08, N = 3SE +/- 0.25, N = 3SE +/- 0.35, N = 3SE +/- 0.29, N = 334.9235.0134.9734.76

Neural Magic DeepSparse

Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: ResNet-50, Baseline - Scenario: Synchronous Single-Streamabcd70140210280350SE +/- 2.76, N = 9SE +/- 3.44, N = 6SE +/- 2.56, N = 12SE +/- 2.94, N = 12338.96337.68338.55338.15

Neural Magic DeepSparse

Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: ResNet-50, Baseline - Scenario: Synchronous Single-Streamabcd0.6661.3321.9982.6643.33SE +/- 0.0255, N = 9SE +/- 0.0315, N = 6SE +/- 0.0242, N = 12SE +/- 0.0282, N = 122.94922.96022.95302.9573

Neural Magic DeepSparse

Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Streamabcd2K4K6K8K10KSE +/- 71.77, N = 13SE +/- 89.80, N = 9SE +/- 101.43, N = 7SE +/- 77.36, N = 1211228.6611169.8511163.6611221.14

Neural Magic DeepSparse

Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Streamabcd1.28622.57243.85865.14486.431SE +/- 0.0393, N = 13SE +/- 0.0484, N = 9SE +/- 0.0544, N = 7SE +/- 0.0425, N = 125.68525.71405.71655.6908

Neural Magic DeepSparse

Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Streamabcd2004006008001000SE +/- 11.12, N = 4SE +/- 9.47, N = 3SE +/- 5.37, N = 3SE +/- 9.64, N = 71030.321027.511043.281032.24

Neural Magic DeepSparse

Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Streamabcd0.21840.43680.65520.87361.092SE +/- 0.0105, N = 4SE +/- 0.0090, N = 3SE +/- 0.0048, N = 3SE +/- 0.0094, N = 70.96810.97060.95560.9667

Neural Magic DeepSparse

Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Streamabcd2004006008001000SE +/- 7.51, N = 3SE +/- 6.37, N = 3SE +/- 6.81, N = 3SE +/- 9.22, N = 3828.04828.65828.45825.08

Neural Magic DeepSparse

Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Streamabcd20406080100SE +/- 0.70, N = 3SE +/- 0.63, N = 3SE +/- 0.64, N = 3SE +/- 0.87, N = 377.1877.0977.1277.46

Neural Magic DeepSparse

Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Streamabcd4080120160200SE +/- 1.29, N = 12SE +/- 1.73, N = 12SE +/- 1.69, N = 7SE +/- 1.52, N = 12190.84190.04189.82190.14

Neural Magic DeepSparse

Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Streamabcd1.18462.36923.55384.73845.923SE +/- 0.0382, N = 12SE +/- 0.0526, N = 12SE +/- 0.0492, N = 7SE +/- 0.0457, N = 125.23825.26175.26515.2589

Neural Magic DeepSparse

Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Streamabcd306090120150SE +/- 0.66, N = 3SE +/- 0.66, N = 3SE +/- 1.19, N = 3SE +/- 1.10, N = 3156.06155.72155.77155.60

Neural Magic DeepSparse

Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Streamabcd90180270360450SE +/- 1.62, N = 3SE +/- 1.91, N = 3SE +/- 2.55, N = 3SE +/- 2.06, N = 3407.28408.63408.97408.16

Neural Magic DeepSparse

Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Streamabcd714212835SE +/- 0.30, N = 6SE +/- 0.27, N = 3SE +/- 0.15, N = 3SE +/- 0.10, N = 329.9430.3930.2929.52

Neural Magic DeepSparse

Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Streamabcd816243240SE +/- 0.34, N = 6SE +/- 0.30, N = 3SE +/- 0.16, N = 3SE +/- 0.12, N = 333.4032.9033.0033.86

Neural Magic DeepSparse

Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Streamabcd400800120016002000SE +/- 22.80, N = 3SE +/- 7.71, N = 3SE +/- 5.61, N = 3SE +/- 21.20, N = 31832.791817.991834.481819.69

Neural Magic DeepSparse

Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Streamabcd816243240SE +/- 0.43, N = 3SE +/- 0.14, N = 3SE +/- 0.11, N = 3SE +/- 0.39, N = 334.8735.1434.8235.10

Neural Magic DeepSparse

Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Streamabcd70140210280350SE +/- 2.66, N = 10SE +/- 2.82, N = 9SE +/- 3.30, N = 6SE +/- 2.24, N = 12339.04338.86338.22339.37

Neural Magic DeepSparse

Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Streamabcd0.6651.331.9952.663.325SE +/- 0.0247, N = 10SE +/- 0.0260, N = 9SE +/- 0.0300, N = 6SE +/- 0.0208, N = 122.94842.95002.95552.9455

Neural Magic DeepSparse

Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Streamabcd2004006008001000SE +/- 8.70, N = 3SE +/- 10.26, N = 3SE +/- 9.10, N = 3SE +/- 10.96, N = 3854.05852.95854.25850.89

Neural Magic DeepSparse

Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Streamabcd20406080100SE +/- 0.77, N = 3SE +/- 0.92, N = 3SE +/- 0.80, N = 3SE +/- 0.97, N = 374.8274.9474.7975.08

Neural Magic DeepSparse

Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Streamabcd4080120160200SE +/- 1.36, N = 12SE +/- 1.53, N = 13SE +/- 1.70, N = 12SE +/- 1.62, N = 12194.25193.60193.09193.82

Neural Magic DeepSparse

Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Streamabcd1.16552.3313.49654.6625.8275SE +/- 0.0389, N = 12SE +/- 0.0445, N = 13SE +/- 0.0500, N = 12SE +/- 0.0471, N = 125.14855.16595.18025.1614

Neural Magic DeepSparse

Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Streamabcd30060090012001500SE +/- 14.83, N = 3SE +/- 14.60, N = 4SE +/- 13.91, N = 3SE +/- 17.38, N = 31231.201231.291231.271226.80

Neural Magic DeepSparse

Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Streamabcd1224364860SE +/- 0.63, N = 3SE +/- 0.60, N = 4SE +/- 0.62, N = 3SE +/- 0.70, N = 351.9351.9351.9252.07

Neural Magic DeepSparse

Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Streamabcd4080120160200SE +/- 1.45, N = 12SE +/- 2.05, N = 12SE +/- 1.76, N = 12SE +/- 1.69, N = 8199.69199.34199.33196.79

Neural Magic DeepSparse

Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Streamabcd1.14322.28643.42964.57285.716SE +/- 0.0391, N = 12SE +/- 0.0573, N = 12SE +/- 0.0484, N = 12SE +/- 0.0457, N = 85.00735.01965.01795.0808

Neural Magic DeepSparse

Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Streamabcd4080120160200SE +/- 1.57, N = 8SE +/- 2.09, N = 3SE +/- 1.83, N = 3SE +/- 1.48, N = 9183.50180.32177.15181.20

Neural Magic DeepSparse

Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Streamabcd80160240320400SE +/- 3.00, N = 8SE +/- 4.19, N = 3SE +/- 3.47, N = 3SE +/- 2.84, N = 9347.66353.69359.89352.22

Neural Magic DeepSparse

Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Streamabcd816243240SE +/- 0.33, N = 7SE +/- 0.34, N = 6SE +/- 0.35, N = 6SE +/- 0.23, N = 1234.9635.1135.0135.00

Neural Magic DeepSparse

Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Streamabcd714212835SE +/- 0.28, N = 7SE +/- 0.29, N = 6SE +/- 0.29, N = 6SE +/- 0.20, N = 1228.5928.4728.5528.56

Neural Magic DeepSparse

Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Streamabcd400800120016002000SE +/- 21.42, N = 3SE +/- 26.46, N = 3SE +/- 21.19, N = 3SE +/- 21.54, N = 31880.771875.901879.841865.02

Neural Magic DeepSparse

Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Streamabcd816243240SE +/- 0.38, N = 3SE +/- 0.48, N = 3SE +/- 0.38, N = 3SE +/- 0.40, N = 333.9834.0833.9934.25

Neural Magic DeepSparse

Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Streamabcd20406080100SE +/- 0.11, N = 3SE +/- 0.30, N = 3SE +/- 0.16, N = 3SE +/- 0.72, N = 1096.4396.3896.5595.70

Neural Magic DeepSparse

Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Streamabcd3691215SE +/- 0.01, N = 3SE +/- 0.03, N = 3SE +/- 0.02, N = 3SE +/- 0.08, N = 1010.3610.3710.3510.45

Neural Magic DeepSparse

Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Streamabcd306090120150SE +/- 0.13, N = 3SE +/- 0.14, N = 3SE +/- 0.19, N = 3SE +/- 0.26, N = 3133.60133.82133.60133.60

Neural Magic DeepSparse

Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Streamabcd100200300400500SE +/- 0.32, N = 3SE +/- 0.39, N = 3SE +/- 0.55, N = 3SE +/- 0.75, N = 3475.13474.37475.35475.45

Neural Magic DeepSparse

Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.6Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Streamabcd816243240SE +/- 0.31, N = 15SE +/- 0.25, N = 15SE +/- 0.33, N = 15SE +/- 0.33, N = 1531.7031.5331.7233.05

Neural Magic DeepSparse

Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.6Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Streamabcd714212835SE +/- 0.31, N = 15SE +/- 0.25, N = 15SE +/- 0.32, N = 15SE +/- 0.31, N = 1531.5831.7331.5630.29

NWChem

Input: C240 Buckyball

OpenBenchmarking.orgSeconds, Fewer Is BetterNWChem 7.0.2Input: C240 Buckyballabcd4008001200160020001744.01730.71757.31748.01. (F9X) gfortran options: -lnwctask -lccsd -lmcscf -lselci -lmp2 -lmoints -lstepper -ldriver -loptim -lnwdft -lgradients -lcphf -lesp -lddscf -ldangchang -lguess -lhessian -lvib -lnwcutil -lrimp2 -lproperty -lsolvation -lnwints -lprepar -lnwmd -lnwpw -lofpw -lpaw -lpspw -lband -lnwpwlib -lcafe -lspace -lanalyze -lqhop -lpfft -ldplot -ldrdy -lvscf -lqmmm -lqmd -letrans -ltce -lbq -lmm -lcons -lperfm -ldntmc -lccca -ldimqm -lga -larmci -lpeigs -l64to32 -lopenblas -lpthread -lrt -llapack -lnwcblas -lmpi_usempif08 -lmpi_mpifh -lmpi -lopen-rte -lopen-pal -lhwloc -levent_core -levent_pthreads -lm -lz -lcomex -m64 -ffast-math -std=legacy -fdefault-integer-8 -finline-functions -O2

Phoronix Test Suite v10.8.5