a Benchmarks for a future article.
Compare your own system(s) to this result file with the
Phoronix Test Suite by running the command:
phoronix-test-suite benchmark 2312143-NE-A8154652071 a Kernel Notes: Transparent Huge Pages: madviseProcessor Notes: Scaling Governor: intel_pstate powersave (EPP: balance_performance) - CPU Microcode: 0x21000161Python Notes: Python 3.11.6Security Notes: gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected
b c d Processor: 2 x INTEL XEON PLATINUM 8592+ @ 3.90GHz (128 Cores / 256 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3B05.TEL4P1 BIOS), Chipset: Intel Device 1bce, Memory: 1008GB, Disk: 3201GB Micron_7450_MTFDKCB3T2TFS, Graphics: ASPEED, Network: 2 x Intel X710 for 10GBASE-T
OS: Ubuntu 23.10, Kernel: 6.5.0-13-generic (x86_64), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080
Kernel Notes: Transparent Huge Pages: madviseCompiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-13-XYspKM/gcc-13-13.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-13-XYspKM/gcc-13-13.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -vProcessor Notes: Scaling Governor: intel_pstate powersave (EPP: balance_performance) - CPU Microcode: 0x21000161Python Notes: Python 3.11.6Security Notes: gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected
a OpenBenchmarking.org Phoronix Test Suite 2 x INTEL XEON PLATINUM 8592+ @ 3.90GHz (128 Cores / 256 Threads) Quanta Cloud S6Q-MB-MPS (3B05.TEL4P1 BIOS) Intel Device 1bce 1008GB 3201GB Micron_7450_MTFDKCB3T2TFS ASPEED 2 x Intel X710 for 10GBASE-T Ubuntu 23.10 6.5.0-13-generic (x86_64) GCC 13.2.0 ext4 1920x1080 Processor Motherboard Chipset Memory Disk Graphics Network OS Kernel Compiler File-System Screen Resolution A Benchmarks System Logs - Transparent Huge Pages: madvise - Scaling Governor: intel_pstate powersave (EPP: balance_performance) - CPU Microcode: 0x21000161 - Python 3.11.6 - gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected - d: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-13-XYspKM/gcc-13-13.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-13-XYspKM/gcc-13-13.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v
a b c d Result Overview Phoronix Test Suite 100% 100% 101% 101% 102% NWChem Neural Magic DeepSparse WRF
a deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Stream deepsparse: ResNet-50, Baseline - Asynchronous Multi-Stream deepsparse: ResNet-50, Baseline - Synchronous Single-Stream deepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Stream deepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Stream deepsparse: ResNet-50, Baseline - Asynchronous Multi-Stream deepsparse: ResNet-50, Baseline - Synchronous Single-Stream deepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Stream deepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream nwchem: C240 Buckyball wrf: conus 2.5km a b c d 133.5159 31.1945 4554.9796 238.6523 1829.9644 338.9562 11228.6625 1030.3181 828.0384 190.8420 156.0563 29.9421 1832.7927 339.0416 854.0525 194.2476 1231.2049 199.6930 183.4966 34.9620 1880.7688 96.4300 133.6023 31.6954 475.6964 32.0613 14.0362 4.2083 34.9169 2.9492 5.6852 0.9681 77.1758 5.2382 407.2777 33.4022 34.8681 2.9484 74.8177 5.1485 51.9301 5.0073 347.6637 28.5876 33.9808 10.3634 475.1265 31.5821 1744 5566.729 133.7189 31.2336 4563.5097 237.0460 1824.3759 337.6799 11169.8517 1027.5117 828.6532 190.0399 155.7222 30.3891 1817.9939 338.8583 852.9455 193.5972 1231.2939 199.3380 180.3157 35.1065 1875.9035 96.3790 133.8239 31.5332 474.1852 32.0368 14.0081 4.2295 35.0063 2.9602 5.7140 0.9706 77.0924 5.2617 408.6263 32.8989 35.1423 2.9500 74.9435 5.1659 51.9273 5.0196 353.6889 28.4701 34.0815 10.3695 474.3687 31.7303 1730.7 5600.976 133.4843 31.6300 4563.4278 230.7640 1827.9780 338.5536 11163.6593 1043.2804 828.4462 189.8175 155.7677 30.2944 1834.4771 338.2192 854.2521 193.0872 1231.2736 199.3323 177.1512 35.0057 1879.8388 96.5474 133.6004 31.7202 475.6804 31.6501 14.0090 4.3436 34.9683 2.9530 5.7165 0.9556 77.1235 5.2651 408.9699 32.9984 34.8217 2.9555 74.7945 5.1802 51.9192 5.0179 359.8919 28.5521 33.9939 10.3507 475.3497 31.5618 1757.3 5583.112 133.0079 32.8936 4556.0233 240.2336 1837.4977 338.1466 11221.1357 1032.2410 825.0841 190.1376 155.6020 29.5231 1819.6861 339.3671 850.8926 193.8186 1226.7960 196.7874 181.2029 34.9987 1865.0193 95.6981 133.5969 33.0540 477.6108 30.4222 14.0315 4.1604 34.7567 2.9573 5.6908 0.9667 77.4573 5.2589 408.1638 33.8635 35.1013 2.9455 75.0848 5.1614 52.0706 5.0808 352.2235 28.5586 34.2508 10.4501 475.4464 30.2891 1748 5617.197 OpenBenchmarking.org
Neural Magic DeepSparse This is a benchmark of Neural Magic's DeepSparse using its built-in deepsparse.benchmark utility and various models from their SparseZoo (https://sparsezoo.neuralmagic.com/). Learn more via the OpenBenchmarking.org test page.
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream a b c d 30 60 90 120 150 SE +/- 0.15, N = 3 SE +/- 0.13, N = 3 SE +/- 0.16, N = 3 SE +/- 0.11, N = 3 133.52 133.72 133.48 133.01
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream a b c d 8 16 24 32 40 SE +/- 0.38, N = 4 SE +/- 0.25, N = 15 SE +/- 0.32, N = 15 SE +/- 0.26, N = 15 31.19 31.23 31.63 32.89
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 1000 2000 3000 4000 5000 SE +/- 49.40, N = 5 SE +/- 40.39, N = 7 SE +/- 43.60, N = 6 SE +/- 42.83, N = 6 4554.98 4563.51 4563.43 4556.02
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 50 100 150 200 250 SE +/- 4.61, N = 15 SE +/- 3.82, N = 15 SE +/- 3.66, N = 15 SE +/- 3.16, N = 3 238.65 237.05 230.76 240.23
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream a b c d 400 800 1200 1600 2000 SE +/- 4.85, N = 3 SE +/- 13.19, N = 3 SE +/- 17.94, N = 3 SE +/- 14.66, N = 3 1829.96 1824.38 1827.98 1837.50
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream a b c d 70 140 210 280 350 SE +/- 2.76, N = 9 SE +/- 3.44, N = 6 SE +/- 2.56, N = 12 SE +/- 2.94, N = 12 338.96 337.68 338.55 338.15
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 2K 4K 6K 8K 10K SE +/- 71.77, N = 13 SE +/- 89.80, N = 9 SE +/- 101.43, N = 7 SE +/- 77.36, N = 12 11228.66 11169.85 11163.66 11221.14
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 200 400 600 800 1000 SE +/- 11.12, N = 4 SE +/- 9.47, N = 3 SE +/- 5.37, N = 3 SE +/- 9.64, N = 7 1030.32 1027.51 1043.28 1032.24
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream a b c d 200 400 600 800 1000 SE +/- 7.51, N = 3 SE +/- 6.37, N = 3 SE +/- 6.81, N = 3 SE +/- 9.22, N = 3 828.04 828.65 828.45 825.08
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream a b c d 40 80 120 160 200 SE +/- 1.29, N = 12 SE +/- 1.73, N = 12 SE +/- 1.69, N = 7 SE +/- 1.52, N = 12 190.84 190.04 189.82 190.14
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream a b c d 30 60 90 120 150 SE +/- 0.66, N = 3 SE +/- 0.66, N = 3 SE +/- 1.19, N = 3 SE +/- 1.10, N = 3 156.06 155.72 155.77 155.60
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream a b c d 7 14 21 28 35 SE +/- 0.30, N = 6 SE +/- 0.27, N = 3 SE +/- 0.15, N = 3 SE +/- 0.10, N = 3 29.94 30.39 30.29 29.52
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream a b c d 400 800 1200 1600 2000 SE +/- 22.80, N = 3 SE +/- 7.71, N = 3 SE +/- 5.61, N = 3 SE +/- 21.20, N = 3 1832.79 1817.99 1834.48 1819.69
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream a b c d 70 140 210 280 350 SE +/- 2.66, N = 10 SE +/- 2.82, N = 9 SE +/- 3.30, N = 6 SE +/- 2.24, N = 12 339.04 338.86 338.22 339.37
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 200 400 600 800 1000 SE +/- 8.70, N = 3 SE +/- 10.26, N = 3 SE +/- 9.10, N = 3 SE +/- 10.96, N = 3 854.05 852.95 854.25 850.89
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 40 80 120 160 200 SE +/- 1.36, N = 12 SE +/- 1.53, N = 13 SE +/- 1.70, N = 12 SE +/- 1.62, N = 12 194.25 193.60 193.09 193.82
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream a b c d 300 600 900 1200 1500 SE +/- 14.83, N = 3 SE +/- 14.60, N = 4 SE +/- 13.91, N = 3 SE +/- 17.38, N = 3 1231.20 1231.29 1231.27 1226.80
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream a b c d 40 80 120 160 200 SE +/- 1.45, N = 12 SE +/- 2.05, N = 12 SE +/- 1.76, N = 12 SE +/- 1.69, N = 8 199.69 199.34 199.33 196.79
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream a b c d 40 80 120 160 200 SE +/- 1.57, N = 8 SE +/- 2.09, N = 3 SE +/- 1.83, N = 3 SE +/- 1.48, N = 9 183.50 180.32 177.15 181.20
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream a b c d 8 16 24 32 40 SE +/- 0.33, N = 7 SE +/- 0.34, N = 6 SE +/- 0.35, N = 6 SE +/- 0.23, N = 12 34.96 35.11 35.01 35.00
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 400 800 1200 1600 2000 SE +/- 21.42, N = 3 SE +/- 26.46, N = 3 SE +/- 21.19, N = 3 SE +/- 21.54, N = 3 1880.77 1875.90 1879.84 1865.02
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 20 40 60 80 100 SE +/- 0.11, N = 3 SE +/- 0.30, N = 3 SE +/- 0.16, N = 3 SE +/- 0.72, N = 10 96.43 96.38 96.55 95.70
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream a b c d 30 60 90 120 150 SE +/- 0.13, N = 3 SE +/- 0.14, N = 3 SE +/- 0.19, N = 3 SE +/- 0.26, N = 3 133.60 133.82 133.60 133.60
OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream a b c d 8 16 24 32 40 SE +/- 0.31, N = 15 SE +/- 0.25, N = 15 SE +/- 0.33, N = 15 SE +/- 0.33, N = 15 31.70 31.53 31.72 33.05
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream a b c d 100 200 300 400 500 SE +/- 0.61, N = 3 SE +/- 0.47, N = 3 SE +/- 0.52, N = 3 SE +/- 0.44, N = 3 475.70 474.19 475.68 477.61
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream a b c d 7 14 21 28 35 SE +/- 0.40, N = 4 SE +/- 0.27, N = 15 SE +/- 0.32, N = 15 SE +/- 0.25, N = 15 32.06 32.04 31.65 30.42
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 4 8 12 16 20 SE +/- 0.16, N = 5 SE +/- 0.13, N = 7 SE +/- 0.14, N = 6 SE +/- 0.14, N = 6 14.04 14.01 14.01 14.03
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 0.9773 1.9546 2.9319 3.9092 4.8865 SE +/- 0.0821, N = 15 SE +/- 0.0666, N = 15 SE +/- 0.0661, N = 15 SE +/- 0.0540, N = 3 4.2083 4.2295 4.3436 4.1604
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream a b c d 8 16 24 32 40 SE +/- 0.08, N = 3 SE +/- 0.25, N = 3 SE +/- 0.35, N = 3 SE +/- 0.29, N = 3 34.92 35.01 34.97 34.76
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream a b c d 0.666 1.332 1.998 2.664 3.33 SE +/- 0.0255, N = 9 SE +/- 0.0315, N = 6 SE +/- 0.0242, N = 12 SE +/- 0.0282, N = 12 2.9492 2.9602 2.9530 2.9573
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 1.2862 2.5724 3.8586 5.1448 6.431 SE +/- 0.0393, N = 13 SE +/- 0.0484, N = 9 SE +/- 0.0544, N = 7 SE +/- 0.0425, N = 12 5.6852 5.7140 5.7165 5.6908
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 0.2184 0.4368 0.6552 0.8736 1.092 SE +/- 0.0105, N = 4 SE +/- 0.0090, N = 3 SE +/- 0.0048, N = 3 SE +/- 0.0094, N = 7 0.9681 0.9706 0.9556 0.9667
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream a b c d 20 40 60 80 100 SE +/- 0.70, N = 3 SE +/- 0.63, N = 3 SE +/- 0.64, N = 3 SE +/- 0.87, N = 3 77.18 77.09 77.12 77.46
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream a b c d 1.1846 2.3692 3.5538 4.7384 5.923 SE +/- 0.0382, N = 12 SE +/- 0.0526, N = 12 SE +/- 0.0492, N = 7 SE +/- 0.0457, N = 12 5.2382 5.2617 5.2651 5.2589
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream a b c d 90 180 270 360 450 SE +/- 1.62, N = 3 SE +/- 1.91, N = 3 SE +/- 2.55, N = 3 SE +/- 2.06, N = 3 407.28 408.63 408.97 408.16
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream a b c d 8 16 24 32 40 SE +/- 0.34, N = 6 SE +/- 0.30, N = 3 SE +/- 0.16, N = 3 SE +/- 0.12, N = 3 33.40 32.90 33.00 33.86
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream a b c d 8 16 24 32 40 SE +/- 0.43, N = 3 SE +/- 0.14, N = 3 SE +/- 0.11, N = 3 SE +/- 0.39, N = 3 34.87 35.14 34.82 35.10
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream a b c d 0.665 1.33 1.995 2.66 3.325 SE +/- 0.0247, N = 10 SE +/- 0.0260, N = 9 SE +/- 0.0300, N = 6 SE +/- 0.0208, N = 12 2.9484 2.9500 2.9555 2.9455
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 20 40 60 80 100 SE +/- 0.77, N = 3 SE +/- 0.92, N = 3 SE +/- 0.80, N = 3 SE +/- 0.97, N = 3 74.82 74.94 74.79 75.08
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 1.1655 2.331 3.4965 4.662 5.8275 SE +/- 0.0389, N = 12 SE +/- 0.0445, N = 13 SE +/- 0.0500, N = 12 SE +/- 0.0471, N = 12 5.1485 5.1659 5.1802 5.1614
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream a b c d 12 24 36 48 60 SE +/- 0.63, N = 3 SE +/- 0.60, N = 4 SE +/- 0.62, N = 3 SE +/- 0.70, N = 3 51.93 51.93 51.92 52.07
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream a b c d 1.1432 2.2864 3.4296 4.5728 5.716 SE +/- 0.0391, N = 12 SE +/- 0.0573, N = 12 SE +/- 0.0484, N = 12 SE +/- 0.0457, N = 8 5.0073 5.0196 5.0179 5.0808
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream a b c d 80 160 240 320 400 SE +/- 3.00, N = 8 SE +/- 4.19, N = 3 SE +/- 3.47, N = 3 SE +/- 2.84, N = 9 347.66 353.69 359.89 352.22
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream a b c d 7 14 21 28 35 SE +/- 0.28, N = 7 SE +/- 0.29, N = 6 SE +/- 0.29, N = 6 SE +/- 0.20, N = 12 28.59 28.47 28.55 28.56
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 8 16 24 32 40 SE +/- 0.38, N = 3 SE +/- 0.48, N = 3 SE +/- 0.38, N = 3 SE +/- 0.40, N = 3 33.98 34.08 33.99 34.25
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 3 6 9 12 15 SE +/- 0.01, N = 3 SE +/- 0.03, N = 3 SE +/- 0.02, N = 3 SE +/- 0.08, N = 10 10.36 10.37 10.35 10.45
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream a b c d 100 200 300 400 500 SE +/- 0.32, N = 3 SE +/- 0.39, N = 3 SE +/- 0.55, N = 3 SE +/- 0.75, N = 3 475.13 474.37 475.35 475.45
OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream a b c d 7 14 21 28 35 SE +/- 0.31, N = 15 SE +/- 0.25, N = 15 SE +/- 0.32, N = 15 SE +/- 0.31, N = 15 31.58 31.73 31.56 30.29
NWChem NWChem is an open-source high performance computational chemistry package. Per NWChem's documentation, "NWChem aims to provide its users with computational chemistry tools that are scalable both in their ability to treat large scientific computational chemistry problems efficiently, and in their use of available parallel computing resources from high-performance parallel supercomputers to conventional workstation clusters." Learn more via the OpenBenchmarking.org test page.
OpenBenchmarking.org Seconds, Fewer Is Better NWChem 7.0.2 Input: C240 Buckyball a b c d 400 800 1200 1600 2000 1744.0 1730.7 1757.3 1748.0 1. (F9X) gfortran options: -lnwctask -lccsd -lmcscf -lselci -lmp2 -lmoints -lstepper -ldriver -loptim -lnwdft -lgradients -lcphf -lesp -lddscf -ldangchang -lguess -lhessian -lvib -lnwcutil -lrimp2 -lproperty -lsolvation -lnwints -lprepar -lnwmd -lnwpw -lofpw -lpaw -lpspw -lband -lnwpwlib -lcafe -lspace -lanalyze -lqhop -lpfft -ldplot -ldrdy -lvscf -lqmmm -lqmd -letrans -ltce -lbq -lmm -lcons -lperfm -ldntmc -lccca -ldimqm -lga -larmci -lpeigs -l64to32 -lopenblas -lpthread -lrt -llapack -lnwcblas -lmpi_usempif08 -lmpi_mpifh -lmpi -lopen-rte -lopen-pal -lhwloc -levent_core -levent_pthreads -lm -lz -lcomex -m64 -ffast-math -std=legacy -fdefault-integer-8 -finline-functions -O2
WRF WRF, the Weather Research and Forecasting Model, is a "next-generation mesoscale numerical weather prediction system designed for both atmospheric research and operational forecasting applications. It features two dynamical cores, a data assimilation system, and a software architecture supporting parallel computation and system extensibility." Learn more via the OpenBenchmarking.org test page.
OpenBenchmarking.org Seconds, Fewer Is Better WRF 4.2.2 Input: conus 2.5km a b c d 1200 2400 3600 4800 6000 5566.73 5600.98 5583.11 5617.20 1. (F9X) gfortran options: -O2 -ftree-vectorize -funroll-loops -ffree-form -fconvert=big-endian -frecord-marker=4 -fallow-invalid-boz -lesmf_time -lwrfio_nf -lnetcdff -lnetcdf -lmpi_usempif08 -lmpi_mpifh -lmpi -lopen-rte -lopen-pal -lhwloc -levent_core -levent_pthreads -lm -lz
a Kernel Notes: Transparent Huge Pages: madviseProcessor Notes: Scaling Governor: intel_pstate powersave (EPP: balance_performance) - CPU Microcode: 0x21000161Python Notes: Python 3.11.6Security Notes: gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected
Testing initiated at 12 December 2023 00:07 by user phoronix.
b Kernel Notes: Transparent Huge Pages: madviseProcessor Notes: Scaling Governor: intel_pstate powersave (EPP: balance_performance) - CPU Microcode: 0x21000161Python Notes: Python 3.11.6Security Notes: gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected
Testing initiated at 12 December 2023 02:28 by user phoronix.
c Kernel Notes: Transparent Huge Pages: madviseProcessor Notes: Scaling Governor: intel_pstate powersave (EPP: balance_performance) - CPU Microcode: 0x21000161Python Notes: Python 3.11.6Security Notes: gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected
Testing initiated at 12 December 2023 10:31 by user phoronix.
d Processor: 2 x INTEL XEON PLATINUM 8592+ @ 3.90GHz (128 Cores / 256 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3B05.TEL4P1 BIOS), Chipset: Intel Device 1bce, Memory: 1008GB, Disk: 3201GB Micron_7450_MTFDKCB3T2TFS, Graphics: ASPEED, Network: 2 x Intel X710 for 10GBASE-T
OS: Ubuntu 23.10, Kernel: 6.5.0-13-generic (x86_64), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080
Kernel Notes: Transparent Huge Pages: madviseCompiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-13-XYspKM/gcc-13-13.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-13-XYspKM/gcc-13-13.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -vProcessor Notes: Scaling Governor: intel_pstate powersave (EPP: balance_performance) - CPU Microcode: 0x21000161Python Notes: Python 3.11.6Security Notes: gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected
Testing initiated at 13 December 2023 00:17 by user phoronix.