AMD EPYC 7763 64-Core testing with a AMD DAYTONA_X (RYM1009B BIOS) and ASPEED on Ubuntu 22.04 via the Phoronix Test Suite.
Compare your own system(s) to this result file with the
Phoronix Test Suite by running the command:
phoronix-test-suite benchmark 2308059-NE-77632204529 7763 2204 - Phoronix Test Suite 7763 2204 AMD EPYC 7763 64-Core testing with a AMD DAYTONA_X (RYM1009B BIOS) and ASPEED on Ubuntu 22.04 via the Phoronix Test Suite.
HTML result view exported from: https://openbenchmarking.org/result/2308059-NE-77632204529&export=pdf&grw&sor .
7763 2204 Processor Motherboard Chipset Memory Disk Graphics Monitor Network OS Kernel Desktop Display Server Vulkan Compiler File-System Screen Resolution a b c AMD EPYC 7763 64-Core @ 2.45GHz (64 Cores / 128 Threads) AMD DAYTONA_X (RYM1009B BIOS) AMD Starship/Matisse 256GB 800GB INTEL SSDPF21Q800GB ASPEED VE228 2 x Mellanox MT27710 Ubuntu 22.04 6.2.0-phx (x86_64) GNOME Shell 42.5 X Server 1.21.1.3 1.3.224 GCC 11.3.0 + LLVM 14.0.0 ext4 1920x1080 OpenBenchmarking.org Kernel Details - Transparent Huge Pages: madvise Compiler Details - --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details - Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa001173 Java Details - OpenJDK Runtime Environment (build 11.0.20+8-post-Ubuntu-1ubuntu122.04) Python Details - Python 3.10.6 Security Details - itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected
7763 2204 brl-cad: VGR Performance Metric deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Asynchronous Multi-Stream deepsparse: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Asynchronous Multi-Stream deepsparse: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Synchronous Single-Stream deepsparse: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Synchronous Single-Stream deepsparse: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Asynchronous Multi-Stream deepsparse: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Asynchronous Multi-Stream deepsparse: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Synchronous Single-Stream deepsparse: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Synchronous Single-Stream deepsparse: ResNet-50, Baseline - Asynchronous Multi-Stream deepsparse: ResNet-50, Baseline - Asynchronous Multi-Stream deepsparse: ResNet-50, Baseline - Synchronous Single-Stream deepsparse: ResNet-50, Baseline - Synchronous Single-Stream deepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Stream deepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Stream deepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Stream deepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2 - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2 - Synchronous Single-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream ncnn: CPU - mobilenet ncnn: CPU-v2-v2 - mobilenet-v2 ncnn: CPU-v3-v3 - mobilenet-v3 ncnn: CPU - shufflenet-v2 ncnn: CPU - mnasnet ncnn: CPU - efficientnet-b0 ncnn: CPU - blazeface ncnn: CPU - googlenet ncnn: CPU - vgg16 ncnn: CPU - resnet18 ncnn: CPU - alexnet ncnn: CPU - resnet50 ncnn: CPU - yolov4-tiny ncnn: CPU - squeezenet_ssd ncnn: CPU - regnety_400m ncnn: CPU - vision_transformer ncnn: CPU - FastestDet build-gcc: Time To Compile blender: BMW27 - CPU-Only blender: Classroom - CPU-Only blender: Fishy Cat - CPU-Only blender: Barbershop - CPU-Only blender: Pabellon Barcelona - CPU-Only vvenc: Bosphorus 4K - Fast vvenc: Bosphorus 4K - Faster vvenc: Bosphorus 1080p - Fast vvenc: Bosphorus 1080p - Faster srsran: Downlink Processor Benchmark srsran: PUSCH Processor Benchmark, Throughput Total srsran: PUSCH Processor Benchmark, Throughput Thread couchdb: 100 - 1000 - 30 couchdb: 100 - 3000 - 30 couchdb: 300 - 1000 - 30 couchdb: 300 - 3000 - 30 couchdb: 500 - 1000 - 30 couchdb: 500 - 3000 - 30 apache-iotdb: 100 - 1 - 200 apache-iotdb: 100 - 1 - 200 apache-iotdb: 100 - 1 - 500 apache-iotdb: 100 - 1 - 500 apache-iotdb: 200 - 1 - 200 apache-iotdb: 200 - 1 - 200 apache-iotdb: 200 - 1 - 500 apache-iotdb: 200 - 1 - 500 apache-iotdb: 500 - 1 - 200 apache-iotdb: 500 - 1 - 200 apache-iotdb: 500 - 1 - 500 apache-iotdb: 500 - 1 - 500 apache-iotdb: 100 - 100 - 200 apache-iotdb: 100 - 100 - 200 apache-iotdb: 100 - 100 - 500 apache-iotdb: 100 - 100 - 500 apache-iotdb: 200 - 100 - 200 apache-iotdb: 200 - 100 - 200 apache-iotdb: 200 - 100 - 500 apache-iotdb: 200 - 100 - 500 apache-iotdb: 500 - 100 - 200 apache-iotdb: 500 - 100 - 200 apache-iotdb: 500 - 100 - 500 apache-iotdb: 500 - 100 - 500 cassandra: Writes a b c 734386 37.6144 840.5416 19.9469 50.1258 1105.3791 28.9150 173.9462 5.747 489.7709 65.2750 86.6104 11.5378 143.0725 223.4715 39.4219 25.356 468.1383 68.2852 159.8492 6.2525 3814.5194 8.3665 723.7149 1.3784 225.3104 141.7036 119.8041 8.3423 46.7204 681.2756 24.4422 40.9048 467.9848 68.3188 159.9660 6.2481 227.4188 140.3979 120.3645 8.3056 326.3123 97.8711 97.3561 10.2645 53.4831 597.9679 28.6310 34.9104 574.9550 55.5817 94.5097 10.5770 165.9980 192.3546 53.7845 18.5858 37.6109 841.4779 20.0720 49.8132 14.11 6.35 7.00 9.09 6.09 9.98 3.97 14.62 23.84 8.50 5.23 15.49 20.66 14.17 35.24 48.79 10.25 1020.133 27.27 68.80 33.70 253.49 84.55 5.991 10.646 16.083 29.352 657.7 9682.1 211.1 101.578 346.085 169.505 572.125 339.967 2390.933 644019.72 17.45 1038515.62 34.36 898967.08 15.24 1232509.19 33.5 1182440.62 13.54 1636128.73 27.1 39287432.92 36.04 51316464.44 81.2 46437377.67 35.09 42048733.22 109.38 51341708.85 35.05 56935634.55 81.81 236650 729876 37.5315 840.9493 20.0279 49.9226 1104.0377 28.9451 172.7505 5.7863 486.1728 65.7359 86.3905 11.5666 143.2058 223.3670 39.5631 25.2659 467.9696 68.2763 159.7481 6.2566 3824.3045 8.3441 732.1101 1.3623 225.4592 141.6185 119.8818 8.3368 46.5739 679.8240 24.5655 40.6993 468.1087 68.2716 159.9238 6.2495 227.6423 140.2607 120.1762 8.3182 326.5439 97.8434 97.6121 10.2380 53.5531 596.8091 28.6329 34.9082 575.2859 55.5601 94.0445 10.6299 166.2231 192.1645 53.7023 18.6144 37.5352 840.2573 20.0315 49.9144 13.97 6.26 6.55 7.93 5.9 9.78 3.43 14.53 23.64 8.42 5.22 15.34 20.51 14.11 27.54 48.49 9.04 1020.846 27.5 68.50 33.76 253.77 84.35 5.993 10.815 16.090 29.39 658.1 9718.6 208.2 648308.27 17.28 1069145.79 32.92 978176.76 13.6 1226219.88 33.86 1365831.5 11.63 1686943.16 26 38401769.17 36.85 50507747.12 82.26 47245476.78 33.84 41987111.39 110.88 50045888.98 36.1 59505306.55 79.83 238161 730434 37.5814 840.1234 20.0245 49.9312 1103.3552 28.9644 172.0659 5.8092 482.1274 66.2837 86.3219 11.5764 143.5714 222.8175 39.5117 25.2989 468.3293 68.2503 160.5932 6.2238 3823.0833 8.3468 731.5099 1.3634 225.8007 141.4775 119.9790 8.3300 46.9021 679.8118 24.5284 40.7606 467.6396 68.3388 160.6985 6.2195 227.5740 140.3036 120.5813 8.2903 326.4108 97.8698 96.8822 10.3146 53.6166 596.5343 28.6416 34.8978 575.1158 55.5835 94.5548 10.5720 166.058 192.2148 53.952 18.528 37.5755 840.4350 20.0536 49.8580 14.03 6.17 6.34 7.60 5.86 9.75 3.48 14.47 23.91 8.51 5.22 15.54 20.80 14.59 27.59 48.43 8.88 1020.216 27.24 68.70 33.72 253.43 84.17 5.976 10.818 16.055 29.469 619.3 9727.1 210.8 667880.96 16.35 1044153.44 34.08 870795.92 16.07 1261385.89 32.71 1367763.49 11.83 1446487.7 31.4 39945212.99 35.01 52464142.83 79.16 46674344.69 34.79 43363203.76 106.73 49201448.81 37.12 56463717.54 83.14 234887 OpenBenchmarking.org
BRL-CAD VGR Performance Metric OpenBenchmarking.org VGR Performance Metric, More Is Better BRL-CAD 7.36 VGR Performance Metric a c b 160K 320K 480K 640K 800K SE +/- 1805.50, N = 2 SE +/- 963.50, N = 2 SE +/- 357.50, N = 2 734386 730434 729876 1. (CXX) g++ options: -std=c++14 -pipe -fvisibility=hidden -fno-strict-aliasing -fno-common -fexceptions -ftemplate-depth-128 -m64 -ggdb3 -O3 -fipa-pta -fstrength-reduce -finline-functions -flto -ltcl8.6 -lregex_brl -lz_brl -lnetpbm -ldl -lm -ltk8.6
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream a c b 9 18 27 36 45 SE +/- 0.09, N = 2 SE +/- 0.02, N = 2 SE +/- 0.01, N = 2 37.61 37.58 37.53
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream c a b 200 400 600 800 1000 SE +/- 0.59, N = 2 SE +/- 1.02, N = 2 SE +/- 0.16, N = 2 840.12 840.54 840.95
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream b c a 5 10 15 20 25 SE +/- 0.04, N = 2 SE +/- 0.02, N = 2 SE +/- 0.01, N = 2 20.03 20.02 19.95
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream b c a 11 22 33 44 55 SE +/- 0.11, N = 2 SE +/- 0.05, N = 2 SE +/- 0.02, N = 2 49.92 49.93 50.13
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c 200 400 600 800 1000 SE +/- 1.07, N = 2 SE +/- 1.15, N = 2 SE +/- 0.21, N = 2 1105.38 1104.04 1103.36
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c 7 14 21 28 35 SE +/- 0.03, N = 2 SE +/- 0.03, N = 2 SE +/- 0.01, N = 2 28.92 28.95 28.96
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream a b c 40 80 120 160 200 SE +/- 0.98, N = 2 SE +/- 0.63, N = 2 SE +/- 0.80, N = 2 173.95 172.75 172.07
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream a b c 1.3071 2.6142 3.9213 5.2284 6.5355 SE +/- 0.0325, N = 2 SE +/- 0.0212, N = 2 SE +/- 0.0271, N = 2 5.7470 5.7863 5.8092
Neural Magic DeepSparse Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Stream a b c 110 220 330 440 550 SE +/- 0.68, N = 2 SE +/- 1.03, N = 2 SE +/- 7.21, N = 2 489.77 486.17 482.13
Neural Magic DeepSparse Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Stream a b c 15 30 45 60 75 SE +/- 0.10, N = 2 SE +/- 0.10, N = 2 SE +/- 0.99, N = 2 65.28 65.74 66.28
Neural Magic DeepSparse Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Stream a b c 20 40 60 80 100 SE +/- 0.59, N = 2 SE +/- 0.08, N = 2 SE +/- 0.42, N = 2 86.61 86.39 86.32
Neural Magic DeepSparse Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Stream a b c 3 6 9 12 15 SE +/- 0.08, N = 2 SE +/- 0.01, N = 2 SE +/- 0.06, N = 2 11.54 11.57 11.58
Neural Magic DeepSparse Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream c b a 30 60 90 120 150 SE +/- 0.03, N = 2 SE +/- 0.18, N = 2 SE +/- 0.01, N = 2 143.57 143.21 143.07
Neural Magic DeepSparse Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream c b a 50 100 150 200 250 SE +/- 0.05, N = 2 SE +/- 0.26, N = 2 SE +/- 0.03, N = 2 222.82 223.37 223.47
Neural Magic DeepSparse Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream b c a 9 18 27 36 45 SE +/- 0.11, N = 2 SE +/- 0.01, N = 2 SE +/- 0.06, N = 2 39.56 39.51 39.42
Neural Magic DeepSparse Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream b c a 6 12 18 24 30 SE +/- 0.07, N = 2 SE +/- 0.01, N = 2 SE +/- 0.04, N = 2 25.27 25.30 25.36
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream c a b 100 200 300 400 500 SE +/- 0.09, N = 2 SE +/- 0.19, N = 2 SE +/- 0.62, N = 2 468.33 468.14 467.97
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream c b a 15 30 45 60 75 SE +/- 0.01, N = 2 SE +/- 0.04, N = 2 SE +/- 0.04, N = 2 68.25 68.28 68.29
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream c a b 40 80 120 160 200 SE +/- 0.80, N = 2 SE +/- 0.43, N = 2 SE +/- 0.22, N = 2 160.59 159.85 159.75
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream c a b 2 4 6 8 10 SE +/- 0.0308, N = 2 SE +/- 0.0166, N = 2 SE +/- 0.0087, N = 2 6.2238 6.2525 6.2566
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream b c a 800 1600 2400 3200 4000 SE +/- 10.54, N = 2 SE +/- 17.23, N = 2 SE +/- 0.54, N = 2 3824.30 3823.08 3814.52
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream b c a 2 4 6 8 10 SE +/- 0.0212, N = 2 SE +/- 0.0366, N = 2 SE +/- 0.0013, N = 2 8.3441 8.3468 8.3665
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream b c a 160 320 480 640 800 SE +/- 2.91, N = 2 SE +/- 0.48, N = 2 SE +/- 10.71, N = 2 732.11 731.51 723.71
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream b c a 0.3101 0.6202 0.9303 1.2404 1.5505 SE +/- 0.0055, N = 2 SE +/- 0.0010, N = 2 SE +/- 0.0205, N = 2 1.3623 1.3634 1.3784
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream c b a 50 100 150 200 250 SE +/- 0.22, N = 2 SE +/- 0.10, N = 2 SE +/- 0.21, N = 2 225.80 225.46 225.31
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream c b a 30 60 90 120 150 SE +/- 0.05, N = 2 SE +/- 0.09, N = 2 SE +/- 0.13, N = 2 141.48 141.62 141.70
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream c b a 30 60 90 120 150 SE +/- 0.01, N = 2 SE +/- 0.02, N = 2 SE +/- 0.08, N = 2 119.98 119.88 119.80
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream c b a 2 4 6 8 10 SE +/- 0.0003, N = 2 SE +/- 0.0012, N = 2 SE +/- 0.0056, N = 2 8.3300 8.3368 8.3423
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream c a b 11 22 33 44 55 SE +/- 0.02, N = 2 SE +/- 0.10, N = 2 SE +/- 0.03, N = 2 46.90 46.72 46.57
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream c b a 150 300 450 600 750 SE +/- 0.34, N = 2 SE +/- 0.17, N = 2 SE +/- 0.37, N = 2 679.81 679.82 681.28
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream b c a 6 12 18 24 30 SE +/- 0.01, N = 2 SE +/- 0.00, N = 2 SE +/- 0.01, N = 2 24.57 24.53 24.44
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream b c a 9 18 27 36 45 SE +/- 0.01, N = 2 SE +/- 0.01, N = 2 SE +/- 0.01, N = 2 40.70 40.76 40.90
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream b a c 100 200 300 400 500 SE +/- 1.23, N = 2 SE +/- 0.36, N = 2 SE +/- 0.26, N = 2 468.11 467.98 467.64
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream b a c 15 30 45 60 75 SE +/- 0.13, N = 2 SE +/- 0.04, N = 2 SE +/- 0.01, N = 2 68.27 68.32 68.34
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream c a b 40 80 120 160 200 SE +/- 0.62, N = 2 SE +/- 0.24, N = 2 SE +/- 0.32, N = 2 160.70 159.97 159.92
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream c a b 2 4 6 8 10 SE +/- 0.0244, N = 2 SE +/- 0.0097, N = 2 SE +/- 0.0121, N = 2 6.2195 6.2481 6.2495
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream b c a 50 100 150 200 250 SE +/- 0.19, N = 2 SE +/- 0.22, N = 2 SE +/- 0.11, N = 2 227.64 227.57 227.42
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream b c a 30 60 90 120 150 SE +/- 0.12, N = 2 SE +/- 0.23, N = 2 SE +/- 0.07, N = 2 140.26 140.30 140.40
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream c a b 30 60 90 120 150 SE +/- 0.08, N = 2 SE +/- 0.19, N = 2 SE +/- 0.02, N = 2 120.58 120.36 120.18
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream c a b 2 4 6 8 10 SE +/- 0.0053, N = 2 SE +/- 0.0127, N = 2 SE +/- 0.0014, N = 2 8.2903 8.3056 8.3182
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream b c a 70 140 210 280 350 SE +/- 0.48, N = 2 SE +/- 0.49, N = 2 SE +/- 0.12, N = 2 326.54 326.41 326.31
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream b c a 20 40 60 80 100 SE +/- 0.15, N = 2 SE +/- 0.16, N = 2 SE +/- 0.00, N = 2 97.84 97.87 97.87
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream b a c 20 40 60 80 100 SE +/- 0.17, N = 2 SE +/- 0.04, N = 2 SE +/- 0.08, N = 2 97.61 97.36 96.88
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream b a c 3 6 9 12 15 SE +/- 0.02, N = 2 SE +/- 0.00, N = 2 SE +/- 0.01, N = 2 10.24 10.26 10.31
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream c b a 12 24 36 48 60 SE +/- 0.00, N = 2 SE +/- 0.05, N = 2 SE +/- 0.00, N = 2 53.62 53.55 53.48
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream c b a 130 260 390 520 650 SE +/- 0.01, N = 2 SE +/- 0.12, N = 2 SE +/- 0.05, N = 2 596.53 596.81 597.97
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream c b a 7 14 21 28 35 SE +/- 0.02, N = 2 SE +/- 0.04, N = 2 SE +/- 0.03, N = 2 28.64 28.63 28.63
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream c b a 8 16 24 32 40 SE +/- 0.03, N = 2 SE +/- 0.05, N = 2 SE +/- 0.03, N = 2 34.90 34.91 34.91
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream b c a 120 240 360 480 600 SE +/- 0.68, N = 2 SE +/- 0.31, N = 2 SE +/- 0.03, N = 2 575.29 575.12 574.96
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream b a c 12 24 36 48 60 SE +/- 0.06, N = 2 SE +/- 0.03, N = 2 SE +/- 0.04, N = 2 55.56 55.58 55.58
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream c a b 20 40 60 80 100 SE +/- 0.18, N = 2 SE +/- 0.05, N = 2 SE +/- 0.64, N = 2 94.55 94.51 94.04
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream c a b 3 6 9 12 15 SE +/- 0.02, N = 2 SE +/- 0.01, N = 2 SE +/- 0.07, N = 2 10.57 10.58 10.63
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream b c a 40 80 120 160 200 SE +/- 0.06, N = 2 SE +/- 0.08, N = 2 SE +/- 0.05, N = 2 166.22 166.06 166.00
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream b c a 40 80 120 160 200 SE +/- 0.00, N = 2 SE +/- 0.21, N = 2 SE +/- 0.06, N = 2 192.16 192.21 192.35
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream c a b 12 24 36 48 60 SE +/- 0.10, N = 2 SE +/- 0.08, N = 2 SE +/- 0.01, N = 2 53.95 53.78 53.70
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream c a b 5 10 15 20 25 SE +/- 0.03, N = 2 SE +/- 0.03, N = 2 SE +/- 0.00, N = 2 18.53 18.59 18.61
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream a c b 9 18 27 36 45 SE +/- 0.04, N = 2 SE +/- 0.02, N = 2 SE +/- 0.01, N = 2 37.61 37.58 37.54
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream b c a 200 400 600 800 1000 SE +/- 0.01, N = 2 SE +/- 0.44, N = 2 SE +/- 0.49, N = 2 840.26 840.44 841.48
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream a c b 5 10 15 20 25 SE +/- 0.02, N = 2 SE +/- 0.02, N = 2 SE +/- 0.00, N = 2 20.07 20.05 20.03
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream a c b 11 22 33 44 55 SE +/- 0.05, N = 2 SE +/- 0.06, N = 2 SE +/- 0.00, N = 2 49.81 49.86 49.91
NCNN Target: CPU - Model: mobilenet OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: mobilenet b c a 4 8 12 16 20 SE +/- 0.03, N = 2 SE +/- 0.09, N = 2 SE +/- 0.04, N = 2 13.97 14.03 14.11 MIN: 13.64 / MAX: 19.68 MIN: 13.68 / MAX: 18.14 MIN: 13.76 / MAX: 19.75 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU-v2-v2 - Model: mobilenet-v2 OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU-v2-v2 - Model: mobilenet-v2 c b a 2 4 6 8 10 SE +/- 0.07, N = 2 SE +/- 0.00, N = 2 SE +/- 0.01, N = 2 6.17 6.26 6.35 MIN: 6.02 / MAX: 6.83 MIN: 6.11 / MAX: 12.76 MIN: 6.19 / MAX: 12.45 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU-v3-v3 - Model: mobilenet-v3 OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU-v3-v3 - Model: mobilenet-v3 c b a 2 4 6 8 10 SE +/- 0.04, N = 2 SE +/- 0.18, N = 2 SE +/- 0.55, N = 2 6.34 6.55 7.00 MIN: 6.15 / MAX: 11.57 MIN: 6.24 / MAX: 7.61 MIN: 6.3 / MAX: 10.2 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: shufflenet-v2 OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: shufflenet-v2 c b a 3 6 9 12 15 SE +/- 0.04, N = 2 SE +/- 0.31, N = 2 SE +/- 1.23, N = 2 7.60 7.93 9.09 MIN: 7.44 / MAX: 11.6 MIN: 7.51 / MAX: 11.45 MIN: 7.74 / MAX: 15.92 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: mnasnet OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: mnasnet c b a 2 4 6 8 10 SE +/- 0.05, N = 2 SE +/- 0.00, N = 2 SE +/- 0.11, N = 2 5.86 5.90 6.09 MIN: 5.73 / MAX: 11.67 MIN: 5.81 / MAX: 12.33 MIN: 5.89 / MAX: 10.35 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: efficientnet-b0 OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: efficientnet-b0 c b a 3 6 9 12 15 SE +/- 0.04, N = 2 SE +/- 0.01, N = 2 SE +/- 0.03, N = 2 9.75 9.78 9.98 MIN: 9.58 / MAX: 13.38 MIN: 9.64 / MAX: 16.01 MIN: 9.82 / MAX: 10.96 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: blazeface OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: blazeface b c a 0.8933 1.7866 2.6799 3.5732 4.4665 SE +/- 0.01, N = 2 SE +/- 0.05, N = 2 SE +/- 0.09, N = 2 3.43 3.48 3.97 MIN: 3.35 / MAX: 3.83 MIN: 3.32 / MAX: 8.58 MIN: 3.5 / MAX: 7.61 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: googlenet OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: googlenet c b a 4 8 12 16 20 SE +/- 0.06, N = 2 SE +/- 0.02, N = 2 SE +/- 0.03, N = 2 14.47 14.53 14.62 MIN: 14.26 / MAX: 20.57 MIN: 14.31 / MAX: 24.11 MIN: 14.46 / MAX: 25.51 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: vgg16 OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: vgg16 b a c 6 12 18 24 30 SE +/- 0.06, N = 2 SE +/- 0.05, N = 2 SE +/- 0.08, N = 2 23.64 23.84 23.91 MIN: 23.33 / MAX: 28.05 MIN: 23.45 / MAX: 28.55 MIN: 23.48 / MAX: 30.63 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: resnet18 OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: resnet18 b a c 2 4 6 8 10 SE +/- 0.04, N = 2 SE +/- 0.03, N = 2 SE +/- 0.03, N = 2 8.42 8.50 8.51 MIN: 8.27 / MAX: 14.6 MIN: 8.33 / MAX: 14.69 MIN: 8.3 / MAX: 13.61 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: alexnet OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: alexnet b c a 1.1768 2.3536 3.5304 4.7072 5.884 SE +/- 0.02, N = 2 SE +/- 0.02, N = 2 SE +/- 0.01, N = 2 5.22 5.22 5.23 MIN: 5.11 / MAX: 5.77 MIN: 5.12 / MAX: 7.84 MIN: 5.12 / MAX: 11.62 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: resnet50 OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: resnet50 b a c 4 8 12 16 20 SE +/- 0.10, N = 2 SE +/- 0.08, N = 2 SE +/- 0.14, N = 2 15.34 15.49 15.54 MIN: 15.07 / MAX: 21.69 MIN: 15.24 / MAX: 21.82 MIN: 15.15 / MAX: 27.2 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: yolov4-tiny OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: yolov4-tiny b a c 5 10 15 20 25 SE +/- 0.08, N = 2 SE +/- 0.02, N = 2 SE +/- 0.13, N = 2 20.51 20.66 20.80 MIN: 19.87 / MAX: 24.86 MIN: 20.04 / MAX: 25.04 MIN: 20.01 / MAX: 96.45 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: squeezenet_ssd OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: squeezenet_ssd b a c 4 8 12 16 20 SE +/- 0.09, N = 2 SE +/- 0.01, N = 2 SE +/- 0.62, N = 2 14.11 14.17 14.59 MIN: 13.32 / MAX: 18.63 MIN: 13.53 / MAX: 18.45 MIN: 13.37 / MAX: 277.61 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: regnety_400m OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: regnety_400m b c a 8 16 24 32 40 SE +/- 0.15, N = 2 SE +/- 0.07, N = 2 SE +/- 5.99, N = 2 27.54 27.59 35.24 MIN: 26.96 / MAX: 33.56 MIN: 26.64 / MAX: 33.41 MIN: 27.86 / MAX: 47.9 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: vision_transformer OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: vision_transformer c b a 11 22 33 44 55 SE +/- 0.30, N = 2 SE +/- 0.04, N = 2 SE +/- 0.07, N = 2 48.43 48.49 48.79 MIN: 47.33 / MAX: 85.35 MIN: 47.44 / MAX: 58.53 MIN: 47.65 / MAX: 78.36 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: FastestDet OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: FastestDet c b a 3 6 9 12 15 SE +/- 0.01, N = 2 SE +/- 0.11, N = 2 SE +/- 0.94, N = 2 8.88 9.04 10.25 MIN: 8.58 / MAX: 13.34 MIN: 8.65 / MAX: 15.19 MIN: 8.95 / MAX: 17.14 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
Timed GCC Compilation Time To Compile OpenBenchmarking.org Seconds, Fewer Is Better Timed GCC Compilation 13.2 Time To Compile a c b 200 400 600 800 1000 SE +/- 1.81, N = 2 SE +/- 0.66, N = 2 SE +/- 0.03, N = 2 1020.13 1020.22 1020.85
Blender Blend File: BMW27 - Compute: CPU-Only OpenBenchmarking.org Seconds, Fewer Is Better Blender 3.6 Blend File: BMW27 - Compute: CPU-Only c a b 6 12 18 24 30 SE +/- 0.04, N = 2 SE +/- 0.06, N = 2 SE +/- 0.15, N = 2 27.24 27.27 27.50
Blender Blend File: Classroom - Compute: CPU-Only OpenBenchmarking.org Seconds, Fewer Is Better Blender 3.6 Blend File: Classroom - Compute: CPU-Only b c a 15 30 45 60 75 SE +/- 0.03, N = 2 SE +/- 0.13, N = 2 SE +/- 0.14, N = 2 68.50 68.70 68.80
Blender Blend File: Fishy Cat - Compute: CPU-Only OpenBenchmarking.org Seconds, Fewer Is Better Blender 3.6 Blend File: Fishy Cat - Compute: CPU-Only a c b 8 16 24 32 40 SE +/- 0.02, N = 2 SE +/- 0.01, N = 2 SE +/- 0.25, N = 2 33.70 33.72 33.76
Blender Blend File: Barbershop - Compute: CPU-Only OpenBenchmarking.org Seconds, Fewer Is Better Blender 3.6 Blend File: Barbershop - Compute: CPU-Only c a b 60 120 180 240 300 SE +/- 0.52, N = 2 SE +/- 0.11, N = 2 SE +/- 0.23, N = 2 253.43 253.49 253.77
Blender Blend File: Pabellon Barcelona - Compute: CPU-Only OpenBenchmarking.org Seconds, Fewer Is Better Blender 3.6 Blend File: Pabellon Barcelona - Compute: CPU-Only c b a 20 40 60 80 100 SE +/- 0.04, N = 2 SE +/- 0.14, N = 2 SE +/- 0.40, N = 2 84.17 84.35 84.55
VVenC Video Input: Bosphorus 4K - Video Preset: Fast OpenBenchmarking.org Frames Per Second, More Is Better VVenC 1.9 Video Input: Bosphorus 4K - Video Preset: Fast b a c 1.3484 2.6968 4.0452 5.3936 6.742 SE +/- 0.006, N = 2 SE +/- 0.001, N = 2 SE +/- 0.002, N = 2 5.993 5.991 5.976 1. (CXX) g++ options: -O3 -flto -fno-fat-lto-objects -flto=auto
VVenC Video Input: Bosphorus 4K - Video Preset: Faster OpenBenchmarking.org Frames Per Second, More Is Better VVenC 1.9 Video Input: Bosphorus 4K - Video Preset: Faster c b a 3 6 9 12 15 SE +/- 0.00, N = 2 SE +/- 0.02, N = 2 SE +/- 0.18, N = 2 10.82 10.82 10.65 1. (CXX) g++ options: -O3 -flto -fno-fat-lto-objects -flto=auto
VVenC Video Input: Bosphorus 1080p - Video Preset: Fast OpenBenchmarking.org Frames Per Second, More Is Better VVenC 1.9 Video Input: Bosphorus 1080p - Video Preset: Fast b a c 4 8 12 16 20 SE +/- 0.02, N = 2 SE +/- 0.02, N = 2 SE +/- 0.02, N = 2 16.09 16.08 16.06 1. (CXX) g++ options: -O3 -flto -fno-fat-lto-objects -flto=auto
VVenC Video Input: Bosphorus 1080p - Video Preset: Faster OpenBenchmarking.org Frames Per Second, More Is Better VVenC 1.9 Video Input: Bosphorus 1080p - Video Preset: Faster c b a 7 14 21 28 35 SE +/- 0.07, N = 2 SE +/- 0.11, N = 2 SE +/- 0.07, N = 2 29.47 29.39 29.35 1. (CXX) g++ options: -O3 -flto -fno-fat-lto-objects -flto=auto
srsRAN Project Test: Downlink Processor Benchmark OpenBenchmarking.org Mbps, More Is Better srsRAN Project 23.5 Test: Downlink Processor Benchmark b a c 140 280 420 560 700 SE +/- 27.75, N = 2 SE +/- 17.85, N = 2 SE +/- 0.35, N = 2 658.1 657.7 619.3 1. (CXX) g++ options: -march=native -mfma -O3 -fno-trapping-math -fno-math-errno -lgtest
srsRAN Project Test: PUSCH Processor Benchmark, Throughput Total OpenBenchmarking.org Mbps, More Is Better srsRAN Project 23.5 Test: PUSCH Processor Benchmark, Throughput Total c b a 2K 4K 6K 8K 10K SE +/- 56.45, N = 2 SE +/- 44.35, N = 2 SE +/- 13.30, N = 2 9727.1 9718.6 9682.1 1. (CXX) g++ options: -march=native -mfma -O3 -fno-trapping-math -fno-math-errno -lgtest
srsRAN Project Test: PUSCH Processor Benchmark, Throughput Thread OpenBenchmarking.org Mbps, More Is Better srsRAN Project 23.5 Test: PUSCH Processor Benchmark, Throughput Thread a c b 50 100 150 200 250 SE +/- 0.10, N = 2 SE +/- 0.20, N = 2 SE +/- 1.90, N = 2 211.1 210.8 208.2 1. (CXX) g++ options: -march=native -mfma -O3 -fno-trapping-math -fno-math-errno -lgtest
Apache CouchDB Bulk Size: 100 - Inserts: 1000 - Rounds: 30 OpenBenchmarking.org Seconds, Fewer Is Better Apache CouchDB 3.3.2 Bulk Size: 100 - Inserts: 1000 - Rounds: 30 a 20 40 60 80 100 SE +/- 0.50, N = 2 101.58 1. (CXX) g++ options: -std=c++17 -lmozjs-78 -lm -lei -fPIC -MMD
Apache CouchDB Bulk Size: 100 - Inserts: 3000 - Rounds: 30 OpenBenchmarking.org Seconds, Fewer Is Better Apache CouchDB 3.3.2 Bulk Size: 100 - Inserts: 3000 - Rounds: 30 a 80 160 240 320 400 SE +/- 0.25, N = 2 346.09 1. (CXX) g++ options: -std=c++17 -lmozjs-78 -lm -lei -fPIC -MMD
Apache CouchDB Bulk Size: 300 - Inserts: 1000 - Rounds: 30 OpenBenchmarking.org Seconds, Fewer Is Better Apache CouchDB 3.3.2 Bulk Size: 300 - Inserts: 1000 - Rounds: 30 a 40 80 120 160 200 SE +/- 0.64, N = 2 169.51 1. (CXX) g++ options: -std=c++17 -lmozjs-78 -lm -lei -fPIC -MMD
Apache CouchDB Bulk Size: 300 - Inserts: 3000 - Rounds: 30 OpenBenchmarking.org Seconds, Fewer Is Better Apache CouchDB 3.3.2 Bulk Size: 300 - Inserts: 3000 - Rounds: 30 a 120 240 360 480 600 SE +/- 0.52, N = 2 572.13 1. (CXX) g++ options: -std=c++17 -lmozjs-78 -lm -lei -fPIC -MMD
Apache CouchDB Bulk Size: 500 - Inserts: 1000 - Rounds: 30 OpenBenchmarking.org Seconds, Fewer Is Better Apache CouchDB 3.3.2 Bulk Size: 500 - Inserts: 1000 - Rounds: 30 a 70 140 210 280 350 SE +/- 8.78, N = 2 339.97 1. (CXX) g++ options: -std=c++17 -lmozjs-78 -lm -lei -fPIC -MMD
Apache CouchDB Bulk Size: 500 - Inserts: 3000 - Rounds: 30 OpenBenchmarking.org Seconds, Fewer Is Better Apache CouchDB 3.3.2 Bulk Size: 500 - Inserts: 3000 - Rounds: 30 a 500 1000 1500 2000 2500 2390.93 1. (CXX) g++ options: -std=c++17 -lmozjs-78 -lm -lei -fPIC -MMD
Apache IoTDB Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 200 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 200 c b a 140K 280K 420K 560K 700K 667880.96 648308.27 644019.72
Apache IoTDB Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 200 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 200 a b c 4 8 12 16 20 17.45 17.28 16.35 MAX: 645.35 MAX: 644.33 MAX: 668.86
Apache IoTDB Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 500 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 500 b c a 200K 400K 600K 800K 1000K 1069145.79 1044153.44 1038515.62
Apache IoTDB Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 500 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 500 a c b 8 16 24 32 40 34.36 34.08 32.92 MAX: 704.53 MAX: 699.28 MAX: 728.63
Apache IoTDB Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 200 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 200 b a c 200K 400K 600K 800K 1000K 978176.76 898967.08 870795.92
Apache IoTDB Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 200 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 200 c a b 4 8 12 16 20 16.07 15.24 13.60 MAX: 592.48 MAX: 583.94 MAX: 586.94
Apache IoTDB Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 500 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 500 c a b 300K 600K 900K 1200K 1500K 1261385.89 1232509.19 1226219.88
Apache IoTDB Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 500 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 500 b a c 8 16 24 32 40 33.86 33.50 32.71 MAX: 659.59 MAX: 690.29 MAX: 725.08
Apache IoTDB Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 200 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 200 c b a 300K 600K 900K 1200K 1500K 1367763.49 1365831.50 1182440.62
Apache IoTDB Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 200 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 200 a c b 3 6 9 12 15 13.54 11.83 11.63 MAX: 856.65 MAX: 836.9 MAX: 860.78
Apache IoTDB Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 500 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 500 b a c 400K 800K 1200K 1600K 2000K 1686943.16 1636128.73 1446487.70
Apache IoTDB Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 500 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 500 c a b 7 14 21 28 35 31.4 27.1 26.0 MAX: 890.05 MAX: 934.45 MAX: 873.88
Apache IoTDB Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 200 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 200 c a b 9M 18M 27M 36M 45M 39945212.99 39287432.92 38401769.17
Apache IoTDB Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 200 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 200 b a c 8 16 24 32 40 36.85 36.04 35.01 MAX: 721.27 MAX: 804.01 MAX: 746.4
Apache IoTDB Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 500 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 500 c a b 11M 22M 33M 44M 55M 52464142.83 51316464.44 50507747.12
Apache IoTDB Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 500 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 500 b a c 20 40 60 80 100 82.26 81.20 79.16 MAX: 864.29 MAX: 1009.28 MAX: 1006.03
Apache IoTDB Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 200 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 200 b c a 10M 20M 30M 40M 50M 47245476.78 46674344.69 46437377.67
Apache IoTDB Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 200 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 200 a c b 8 16 24 32 40 35.09 34.79 33.84 MAX: 804.64 MAX: 780.01 MAX: 773.52
Apache IoTDB Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 500 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 500 c a b 9M 18M 27M 36M 45M 43363203.76 42048733.22 41987111.39
Apache IoTDB Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 500 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 500 b a c 20 40 60 80 100 110.88 109.38 106.73 MAX: 3569.78 MAX: 3597.09 MAX: 3485.91
Apache IoTDB Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 200 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 200 a b c 11M 22M 33M 44M 55M 51341708.85 50045888.98 49201448.81
Apache IoTDB Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 200 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 200 c b a 9 18 27 36 45 37.12 36.10 35.05 MAX: 2182.81 MAX: 1990.15 MAX: 2157.23
Apache IoTDB Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 500 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 500 b a c 13M 26M 39M 52M 65M 59505306.55 56935634.55 56463717.54
Apache IoTDB Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 500 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 500 c a b 20 40 60 80 100 83.14 81.81 79.83 MAX: 2932.1 MAX: 3018.16 MAX: 1607.86
Apache Cassandra Test: Writes OpenBenchmarking.org Op/s, More Is Better Apache Cassandra 4.1.3 Test: Writes b a c 50K 100K 150K 200K 250K SE +/- 817.50, N = 2 SE +/- 633.50, N = 2 SE +/- 669.00, N = 2 238161 236650 234887
Phoronix Test Suite v10.8.4