AMD EPYC 7763 64-Core testing with a AMD DAYTONA_X (RYM1009B BIOS) and ASPEED on Ubuntu 22.04 via the Phoronix Test Suite.
Compare your own system(s) to this result file with the
Phoronix Test Suite by running the command:
phoronix-test-suite benchmark 2308059-NE-77632204529 7763 2204 - Phoronix Test Suite 7763 2204 AMD EPYC 7763 64-Core testing with a AMD DAYTONA_X (RYM1009B BIOS) and ASPEED on Ubuntu 22.04 via the Phoronix Test Suite.
HTML result view exported from: https://openbenchmarking.org/result/2308059-NE-77632204529&sor&gru .
7763 2204 Processor Motherboard Chipset Memory Disk Graphics Monitor Network OS Kernel Desktop Display Server Vulkan Compiler File-System Screen Resolution a b c AMD EPYC 7763 64-Core @ 2.45GHz (64 Cores / 128 Threads) AMD DAYTONA_X (RYM1009B BIOS) AMD Starship/Matisse 256GB 800GB INTEL SSDPF21Q800GB ASPEED VE228 2 x Mellanox MT27710 Ubuntu 22.04 6.2.0-phx (x86_64) GNOME Shell 42.5 X Server 1.21.1.3 1.3.224 GCC 11.3.0 + LLVM 14.0.0 ext4 1920x1080 OpenBenchmarking.org Kernel Details - Transparent Huge Pages: madvise Compiler Details - --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details - Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa001173 Java Details - OpenJDK Runtime Environment (build 11.0.20+8-post-Ubuntu-1ubuntu122.04) Python Details - Python 3.10.6 Security Details - itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected
7763 2204 apache-iotdb: 100 - 1 - 200 apache-iotdb: 100 - 1 - 500 apache-iotdb: 200 - 1 - 200 apache-iotdb: 200 - 1 - 500 apache-iotdb: 500 - 1 - 200 apache-iotdb: 500 - 1 - 500 apache-iotdb: 100 - 100 - 200 apache-iotdb: 100 - 100 - 500 apache-iotdb: 200 - 100 - 200 apache-iotdb: 200 - 100 - 500 apache-iotdb: 500 - 100 - 200 apache-iotdb: 500 - 100 - 500 vvenc: Bosphorus 4K - Fast vvenc: Bosphorus 4K - Faster vvenc: Bosphorus 1080p - Fast vvenc: Bosphorus 1080p - Faster deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Asynchronous Multi-Stream deepsparse: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Synchronous Single-Stream deepsparse: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Asynchronous Multi-Stream deepsparse: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Synchronous Single-Stream deepsparse: ResNet-50, Baseline - Asynchronous Multi-Stream deepsparse: ResNet-50, Baseline - Synchronous Single-Stream deepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Stream deepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2 - Synchronous Single-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream srsran: Downlink Processor Benchmark srsran: PUSCH Processor Benchmark, Throughput Total srsran: PUSCH Processor Benchmark, Throughput Thread cassandra: Writes apache-iotdb: 100 - 1 - 200 apache-iotdb: 100 - 1 - 500 apache-iotdb: 200 - 1 - 200 apache-iotdb: 200 - 1 - 500 apache-iotdb: 500 - 1 - 200 apache-iotdb: 500 - 1 - 500 apache-iotdb: 100 - 100 - 200 apache-iotdb: 100 - 100 - 500 apache-iotdb: 200 - 100 - 200 apache-iotdb: 200 - 100 - 500 apache-iotdb: 500 - 100 - 200 apache-iotdb: 500 - 100 - 500 brl-cad: VGR Performance Metric ncnn: CPU - mobilenet ncnn: CPU-v2-v2 - mobilenet-v2 ncnn: CPU-v3-v3 - mobilenet-v3 ncnn: CPU - shufflenet-v2 ncnn: CPU - mnasnet ncnn: CPU - efficientnet-b0 ncnn: CPU - blazeface ncnn: CPU - googlenet ncnn: CPU - vgg16 ncnn: CPU - resnet18 ncnn: CPU - alexnet ncnn: CPU - resnet50 ncnn: CPU - yolov4-tiny ncnn: CPU - squeezenet_ssd ncnn: CPU - regnety_400m ncnn: CPU - vision_transformer ncnn: CPU - FastestDet deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Asynchronous Multi-Stream deepsparse: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Synchronous Single-Stream deepsparse: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Asynchronous Multi-Stream deepsparse: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Synchronous Single-Stream deepsparse: ResNet-50, Baseline - Asynchronous Multi-Stream deepsparse: ResNet-50, Baseline - Synchronous Single-Stream deepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Stream deepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2 - Synchronous Single-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream build-gcc: Time To Compile couchdb: 100 - 1000 - 30 couchdb: 100 - 3000 - 30 couchdb: 300 - 1000 - 30 couchdb: 300 - 3000 - 30 couchdb: 500 - 1000 - 30 couchdb: 500 - 3000 - 30 blender: BMW27 - CPU-Only blender: Classroom - CPU-Only blender: Fishy Cat - CPU-Only blender: Barbershop - CPU-Only blender: Pabellon Barcelona - CPU-Only a b c 17.45 34.36 15.24 33.5 13.54 27.1 36.04 81.2 35.09 109.38 35.05 81.81 5.991 10.646 16.083 29.352 37.6144 19.9469 1105.3791 173.9462 489.7709 86.6104 143.0725 39.4219 468.1383 159.8492 3814.5194 723.7149 225.3104 119.8041 46.7204 24.4422 467.9848 159.9660 227.4188 120.3645 326.3123 97.3561 53.4831 28.6310 574.9550 94.5097 165.9980 53.7845 37.6109 20.0720 657.7 9682.1 211.1 236650 644019.72 1038515.62 898967.08 1232509.19 1182440.62 1636128.73 39287432.92 51316464.44 46437377.67 42048733.22 51341708.85 56935634.55 734386 14.11 6.35 7.00 9.09 6.09 9.98 3.97 14.62 23.84 8.50 5.23 15.49 20.66 14.17 35.24 48.79 10.25 840.5416 50.1258 28.9150 5.747 65.2750 11.5378 223.4715 25.356 68.2852 6.2525 8.3665 1.3784 141.7036 8.3423 681.2756 40.9048 68.3188 6.2481 140.3979 8.3056 97.8711 10.2645 597.9679 34.9104 55.5817 10.5770 192.3546 18.5858 841.4779 49.8132 1020.133 101.578 346.085 169.505 572.125 339.967 2390.933 27.27 68.80 33.70 253.49 84.55 17.28 32.92 13.6 33.86 11.63 26 36.85 82.26 33.84 110.88 36.1 79.83 5.993 10.815 16.090 29.39 37.5315 20.0279 1104.0377 172.7505 486.1728 86.3905 143.2058 39.5631 467.9696 159.7481 3824.3045 732.1101 225.4592 119.8818 46.5739 24.5655 468.1087 159.9238 227.6423 120.1762 326.5439 97.6121 53.5531 28.6329 575.2859 94.0445 166.2231 53.7023 37.5352 20.0315 658.1 9718.6 208.2 238161 648308.27 1069145.79 978176.76 1226219.88 1365831.5 1686943.16 38401769.17 50507747.12 47245476.78 41987111.39 50045888.98 59505306.55 729876 13.97 6.26 6.55 7.93 5.9 9.78 3.43 14.53 23.64 8.42 5.22 15.34 20.51 14.11 27.54 48.49 9.04 840.9493 49.9226 28.9451 5.7863 65.7359 11.5666 223.3670 25.2659 68.2763 6.2566 8.3441 1.3623 141.6185 8.3368 679.8240 40.6993 68.2716 6.2495 140.2607 8.3182 97.8434 10.2380 596.8091 34.9082 55.5601 10.6299 192.1645 18.6144 840.2573 49.9144 1020.846 27.5 68.50 33.76 253.77 84.35 16.35 34.08 16.07 32.71 11.83 31.4 35.01 79.16 34.79 106.73 37.12 83.14 5.976 10.818 16.055 29.469 37.5814 20.0245 1103.3552 172.0659 482.1274 86.3219 143.5714 39.5117 468.3293 160.5932 3823.0833 731.5099 225.8007 119.9790 46.9021 24.5284 467.6396 160.6985 227.5740 120.5813 326.4108 96.8822 53.6166 28.6416 575.1158 94.5548 166.058 53.952 37.5755 20.0536 619.3 9727.1 210.8 234887 667880.96 1044153.44 870795.92 1261385.89 1367763.49 1446487.7 39945212.99 52464142.83 46674344.69 43363203.76 49201448.81 56463717.54 730434 14.03 6.17 6.34 7.60 5.86 9.75 3.48 14.47 23.91 8.51 5.22 15.54 20.80 14.59 27.59 48.43 8.88 840.1234 49.9312 28.9644 5.8092 66.2837 11.5764 222.8175 25.2989 68.2503 6.2238 8.3468 1.3634 141.4775 8.3300 679.8118 40.7606 68.3388 6.2195 140.3036 8.2903 97.8698 10.3146 596.5343 34.8978 55.5835 10.5720 192.2148 18.528 840.4350 49.8580 1020.216 27.24 68.70 33.72 253.43 84.17 OpenBenchmarking.org
Apache IoTDB Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 200 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 200 a b c 4 8 12 16 20 17.45 17.28 16.35 MAX: 645.35 MAX: 644.33 MAX: 668.86
Apache IoTDB Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 500 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 500 a c b 8 16 24 32 40 34.36 34.08 32.92 MAX: 704.53 MAX: 699.28 MAX: 728.63
Apache IoTDB Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 200 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 200 c a b 4 8 12 16 20 16.07 15.24 13.60 MAX: 592.48 MAX: 583.94 MAX: 586.94
Apache IoTDB Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 500 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 500 b a c 8 16 24 32 40 33.86 33.50 32.71 MAX: 659.59 MAX: 690.29 MAX: 725.08
Apache IoTDB Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 200 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 200 a c b 3 6 9 12 15 13.54 11.83 11.63 MAX: 856.65 MAX: 836.9 MAX: 860.78
Apache IoTDB Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 500 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 500 c a b 7 14 21 28 35 31.4 27.1 26.0 MAX: 890.05 MAX: 934.45 MAX: 873.88
Apache IoTDB Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 200 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 200 b a c 8 16 24 32 40 36.85 36.04 35.01 MAX: 721.27 MAX: 804.01 MAX: 746.4
Apache IoTDB Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 500 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 500 b a c 20 40 60 80 100 82.26 81.20 79.16 MAX: 864.29 MAX: 1009.28 MAX: 1006.03
Apache IoTDB Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 200 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 200 a c b 8 16 24 32 40 35.09 34.79 33.84 MAX: 804.64 MAX: 780.01 MAX: 773.52
Apache IoTDB Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 500 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 500 b a c 20 40 60 80 100 110.88 109.38 106.73 MAX: 3569.78 MAX: 3597.09 MAX: 3485.91
Apache IoTDB Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 200 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 200 c b a 9 18 27 36 45 37.12 36.10 35.05 MAX: 2182.81 MAX: 1990.15 MAX: 2157.23
Apache IoTDB Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 500 OpenBenchmarking.org Average Latency, More Is Better Apache IoTDB 1.1.2 Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 500 c a b 20 40 60 80 100 83.14 81.81 79.83 MAX: 2932.1 MAX: 3018.16 MAX: 1607.86
VVenC Video Input: Bosphorus 4K - Video Preset: Fast OpenBenchmarking.org Frames Per Second, More Is Better VVenC 1.9 Video Input: Bosphorus 4K - Video Preset: Fast b a c 1.3484 2.6968 4.0452 5.3936 6.742 SE +/- 0.006, N = 2 SE +/- 0.001, N = 2 SE +/- 0.002, N = 2 5.993 5.991 5.976 1. (CXX) g++ options: -O3 -flto -fno-fat-lto-objects -flto=auto
VVenC Video Input: Bosphorus 4K - Video Preset: Faster OpenBenchmarking.org Frames Per Second, More Is Better VVenC 1.9 Video Input: Bosphorus 4K - Video Preset: Faster c b a 3 6 9 12 15 SE +/- 0.00, N = 2 SE +/- 0.02, N = 2 SE +/- 0.18, N = 2 10.82 10.82 10.65 1. (CXX) g++ options: -O3 -flto -fno-fat-lto-objects -flto=auto
VVenC Video Input: Bosphorus 1080p - Video Preset: Fast OpenBenchmarking.org Frames Per Second, More Is Better VVenC 1.9 Video Input: Bosphorus 1080p - Video Preset: Fast b a c 4 8 12 16 20 SE +/- 0.02, N = 2 SE +/- 0.02, N = 2 SE +/- 0.02, N = 2 16.09 16.08 16.06 1. (CXX) g++ options: -O3 -flto -fno-fat-lto-objects -flto=auto
VVenC Video Input: Bosphorus 1080p - Video Preset: Faster OpenBenchmarking.org Frames Per Second, More Is Better VVenC 1.9 Video Input: Bosphorus 1080p - Video Preset: Faster c b a 7 14 21 28 35 SE +/- 0.07, N = 2 SE +/- 0.11, N = 2 SE +/- 0.07, N = 2 29.47 29.39 29.35 1. (CXX) g++ options: -O3 -flto -fno-fat-lto-objects -flto=auto
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream a c b 9 18 27 36 45 SE +/- 0.09, N = 2 SE +/- 0.02, N = 2 SE +/- 0.01, N = 2 37.61 37.58 37.53
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream b c a 5 10 15 20 25 SE +/- 0.04, N = 2 SE +/- 0.02, N = 2 SE +/- 0.01, N = 2 20.03 20.02 19.95
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c 200 400 600 800 1000 SE +/- 1.07, N = 2 SE +/- 1.15, N = 2 SE +/- 0.21, N = 2 1105.38 1104.04 1103.36
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream a b c 40 80 120 160 200 SE +/- 0.98, N = 2 SE +/- 0.63, N = 2 SE +/- 0.80, N = 2 173.95 172.75 172.07
Neural Magic DeepSparse Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Stream a b c 110 220 330 440 550 SE +/- 0.68, N = 2 SE +/- 1.03, N = 2 SE +/- 7.21, N = 2 489.77 486.17 482.13
Neural Magic DeepSparse Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Stream a b c 20 40 60 80 100 SE +/- 0.59, N = 2 SE +/- 0.08, N = 2 SE +/- 0.42, N = 2 86.61 86.39 86.32
Neural Magic DeepSparse Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream c b a 30 60 90 120 150 SE +/- 0.03, N = 2 SE +/- 0.18, N = 2 SE +/- 0.01, N = 2 143.57 143.21 143.07
Neural Magic DeepSparse Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream b c a 9 18 27 36 45 SE +/- 0.11, N = 2 SE +/- 0.01, N = 2 SE +/- 0.06, N = 2 39.56 39.51 39.42
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream c a b 100 200 300 400 500 SE +/- 0.09, N = 2 SE +/- 0.19, N = 2 SE +/- 0.62, N = 2 468.33 468.14 467.97
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream c a b 40 80 120 160 200 SE +/- 0.80, N = 2 SE +/- 0.43, N = 2 SE +/- 0.22, N = 2 160.59 159.85 159.75
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream b c a 800 1600 2400 3200 4000 SE +/- 10.54, N = 2 SE +/- 17.23, N = 2 SE +/- 0.54, N = 2 3824.30 3823.08 3814.52
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream b c a 160 320 480 640 800 SE +/- 2.91, N = 2 SE +/- 0.48, N = 2 SE +/- 10.71, N = 2 732.11 731.51 723.71
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream c b a 50 100 150 200 250 SE +/- 0.22, N = 2 SE +/- 0.10, N = 2 SE +/- 0.21, N = 2 225.80 225.46 225.31
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream c b a 30 60 90 120 150 SE +/- 0.01, N = 2 SE +/- 0.02, N = 2 SE +/- 0.08, N = 2 119.98 119.88 119.80
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream c a b 11 22 33 44 55 SE +/- 0.02, N = 2 SE +/- 0.10, N = 2 SE +/- 0.03, N = 2 46.90 46.72 46.57
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream b c a 6 12 18 24 30 SE +/- 0.01, N = 2 SE +/- 0.00, N = 2 SE +/- 0.01, N = 2 24.57 24.53 24.44
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream b a c 100 200 300 400 500 SE +/- 1.23, N = 2 SE +/- 0.36, N = 2 SE +/- 0.26, N = 2 468.11 467.98 467.64
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream c a b 40 80 120 160 200 SE +/- 0.62, N = 2 SE +/- 0.24, N = 2 SE +/- 0.32, N = 2 160.70 159.97 159.92
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream b c a 50 100 150 200 250 SE +/- 0.19, N = 2 SE +/- 0.22, N = 2 SE +/- 0.11, N = 2 227.64 227.57 227.42
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream c a b 30 60 90 120 150 SE +/- 0.08, N = 2 SE +/- 0.19, N = 2 SE +/- 0.02, N = 2 120.58 120.36 120.18
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream b c a 70 140 210 280 350 SE +/- 0.48, N = 2 SE +/- 0.49, N = 2 SE +/- 0.12, N = 2 326.54 326.41 326.31
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream b a c 20 40 60 80 100 SE +/- 0.17, N = 2 SE +/- 0.04, N = 2 SE +/- 0.08, N = 2 97.61 97.36 96.88
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream c b a 12 24 36 48 60 SE +/- 0.00, N = 2 SE +/- 0.05, N = 2 SE +/- 0.00, N = 2 53.62 53.55 53.48
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream c b a 7 14 21 28 35 SE +/- 0.02, N = 2 SE +/- 0.04, N = 2 SE +/- 0.03, N = 2 28.64 28.63 28.63
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream b c a 120 240 360 480 600 SE +/- 0.68, N = 2 SE +/- 0.31, N = 2 SE +/- 0.03, N = 2 575.29 575.12 574.96
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream c a b 20 40 60 80 100 SE +/- 0.18, N = 2 SE +/- 0.05, N = 2 SE +/- 0.64, N = 2 94.55 94.51 94.04
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream b c a 40 80 120 160 200 SE +/- 0.06, N = 2 SE +/- 0.08, N = 2 SE +/- 0.05, N = 2 166.22 166.06 166.00
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream c a b 12 24 36 48 60 SE +/- 0.10, N = 2 SE +/- 0.08, N = 2 SE +/- 0.01, N = 2 53.95 53.78 53.70
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream a c b 9 18 27 36 45 SE +/- 0.04, N = 2 SE +/- 0.02, N = 2 SE +/- 0.01, N = 2 37.61 37.58 37.54
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.5 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream a c b 5 10 15 20 25 SE +/- 0.02, N = 2 SE +/- 0.02, N = 2 SE +/- 0.00, N = 2 20.07 20.05 20.03
srsRAN Project Test: Downlink Processor Benchmark OpenBenchmarking.org Mbps, More Is Better srsRAN Project 23.5 Test: Downlink Processor Benchmark b a c 140 280 420 560 700 SE +/- 27.75, N = 2 SE +/- 17.85, N = 2 SE +/- 0.35, N = 2 658.1 657.7 619.3 1. (CXX) g++ options: -march=native -mfma -O3 -fno-trapping-math -fno-math-errno -lgtest
srsRAN Project Test: PUSCH Processor Benchmark, Throughput Total OpenBenchmarking.org Mbps, More Is Better srsRAN Project 23.5 Test: PUSCH Processor Benchmark, Throughput Total c b a 2K 4K 6K 8K 10K SE +/- 56.45, N = 2 SE +/- 44.35, N = 2 SE +/- 13.30, N = 2 9727.1 9718.6 9682.1 1. (CXX) g++ options: -march=native -mfma -O3 -fno-trapping-math -fno-math-errno -lgtest
srsRAN Project Test: PUSCH Processor Benchmark, Throughput Thread OpenBenchmarking.org Mbps, More Is Better srsRAN Project 23.5 Test: PUSCH Processor Benchmark, Throughput Thread a c b 50 100 150 200 250 SE +/- 0.10, N = 2 SE +/- 0.20, N = 2 SE +/- 1.90, N = 2 211.1 210.8 208.2 1. (CXX) g++ options: -march=native -mfma -O3 -fno-trapping-math -fno-math-errno -lgtest
Apache Cassandra Test: Writes OpenBenchmarking.org Op/s, More Is Better Apache Cassandra 4.1.3 Test: Writes b a c 50K 100K 150K 200K 250K SE +/- 817.50, N = 2 SE +/- 633.50, N = 2 SE +/- 669.00, N = 2 238161 236650 234887
Apache IoTDB Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 200 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 200 c b a 140K 280K 420K 560K 700K 667880.96 648308.27 644019.72
Apache IoTDB Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 500 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 500 b c a 200K 400K 600K 800K 1000K 1069145.79 1044153.44 1038515.62
Apache IoTDB Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 200 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 200 b a c 200K 400K 600K 800K 1000K 978176.76 898967.08 870795.92
Apache IoTDB Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 500 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 500 c a b 300K 600K 900K 1200K 1500K 1261385.89 1232509.19 1226219.88
Apache IoTDB Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 200 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 200 c b a 300K 600K 900K 1200K 1500K 1367763.49 1365831.50 1182440.62
Apache IoTDB Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 500 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 500 b a c 400K 800K 1200K 1600K 2000K 1686943.16 1636128.73 1446487.70
Apache IoTDB Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 200 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 200 c a b 9M 18M 27M 36M 45M 39945212.99 39287432.92 38401769.17
Apache IoTDB Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 500 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 500 c a b 11M 22M 33M 44M 55M 52464142.83 51316464.44 50507747.12
Apache IoTDB Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 200 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 200 b c a 10M 20M 30M 40M 50M 47245476.78 46674344.69 46437377.67
Apache IoTDB Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 500 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 500 c a b 9M 18M 27M 36M 45M 43363203.76 42048733.22 41987111.39
Apache IoTDB Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 200 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 200 a b c 11M 22M 33M 44M 55M 51341708.85 50045888.98 49201448.81
Apache IoTDB Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 500 OpenBenchmarking.org point/sec, More Is Better Apache IoTDB 1.1.2 Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 500 b a c 13M 26M 39M 52M 65M 59505306.55 56935634.55 56463717.54
BRL-CAD VGR Performance Metric OpenBenchmarking.org VGR Performance Metric, More Is Better BRL-CAD 7.36 VGR Performance Metric a c b 160K 320K 480K 640K 800K SE +/- 1805.50, N = 2 SE +/- 963.50, N = 2 SE +/- 357.50, N = 2 734386 730434 729876 1. (CXX) g++ options: -std=c++14 -pipe -fvisibility=hidden -fno-strict-aliasing -fno-common -fexceptions -ftemplate-depth-128 -m64 -ggdb3 -O3 -fipa-pta -fstrength-reduce -finline-functions -flto -ltcl8.6 -lregex_brl -lz_brl -lnetpbm -ldl -lm -ltk8.6
NCNN Target: CPU - Model: mobilenet OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: mobilenet b c a 4 8 12 16 20 SE +/- 0.03, N = 2 SE +/- 0.09, N = 2 SE +/- 0.04, N = 2 13.97 14.03 14.11 MIN: 13.64 / MAX: 19.68 MIN: 13.68 / MAX: 18.14 MIN: 13.76 / MAX: 19.75 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU-v2-v2 - Model: mobilenet-v2 OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU-v2-v2 - Model: mobilenet-v2 c b a 2 4 6 8 10 SE +/- 0.07, N = 2 SE +/- 0.00, N = 2 SE +/- 0.01, N = 2 6.17 6.26 6.35 MIN: 6.02 / MAX: 6.83 MIN: 6.11 / MAX: 12.76 MIN: 6.19 / MAX: 12.45 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU-v3-v3 - Model: mobilenet-v3 OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU-v3-v3 - Model: mobilenet-v3 c b a 2 4 6 8 10 SE +/- 0.04, N = 2 SE +/- 0.18, N = 2 SE +/- 0.55, N = 2 6.34 6.55 7.00 MIN: 6.15 / MAX: 11.57 MIN: 6.24 / MAX: 7.61 MIN: 6.3 / MAX: 10.2 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: shufflenet-v2 OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: shufflenet-v2 c b a 3 6 9 12 15 SE +/- 0.04, N = 2 SE +/- 0.31, N = 2 SE +/- 1.23, N = 2 7.60 7.93 9.09 MIN: 7.44 / MAX: 11.6 MIN: 7.51 / MAX: 11.45 MIN: 7.74 / MAX: 15.92 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: mnasnet OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: mnasnet c b a 2 4 6 8 10 SE +/- 0.05, N = 2 SE +/- 0.00, N = 2 SE +/- 0.11, N = 2 5.86 5.90 6.09 MIN: 5.73 / MAX: 11.67 MIN: 5.81 / MAX: 12.33 MIN: 5.89 / MAX: 10.35 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: efficientnet-b0 OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: efficientnet-b0 c b a 3 6 9 12 15 SE +/- 0.04, N = 2 SE +/- 0.01, N = 2 SE +/- 0.03, N = 2 9.75 9.78 9.98 MIN: 9.58 / MAX: 13.38 MIN: 9.64 / MAX: 16.01 MIN: 9.82 / MAX: 10.96 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: blazeface OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: blazeface b c a 0.8933 1.7866 2.6799 3.5732 4.4665 SE +/- 0.01, N = 2 SE +/- 0.05, N = 2 SE +/- 0.09, N = 2 3.43 3.48 3.97 MIN: 3.35 / MAX: 3.83 MIN: 3.32 / MAX: 8.58 MIN: 3.5 / MAX: 7.61 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: googlenet OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: googlenet c b a 4 8 12 16 20 SE +/- 0.06, N = 2 SE +/- 0.02, N = 2 SE +/- 0.03, N = 2 14.47 14.53 14.62 MIN: 14.26 / MAX: 20.57 MIN: 14.31 / MAX: 24.11 MIN: 14.46 / MAX: 25.51 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: vgg16 OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: vgg16 b a c 6 12 18 24 30 SE +/- 0.06, N = 2 SE +/- 0.05, N = 2 SE +/- 0.08, N = 2 23.64 23.84 23.91 MIN: 23.33 / MAX: 28.05 MIN: 23.45 / MAX: 28.55 MIN: 23.48 / MAX: 30.63 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: resnet18 OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: resnet18 b a c 2 4 6 8 10 SE +/- 0.04, N = 2 SE +/- 0.03, N = 2 SE +/- 0.03, N = 2 8.42 8.50 8.51 MIN: 8.27 / MAX: 14.6 MIN: 8.33 / MAX: 14.69 MIN: 8.3 / MAX: 13.61 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: alexnet OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: alexnet b c a 1.1768 2.3536 3.5304 4.7072 5.884 SE +/- 0.02, N = 2 SE +/- 0.02, N = 2 SE +/- 0.01, N = 2 5.22 5.22 5.23 MIN: 5.11 / MAX: 5.77 MIN: 5.12 / MAX: 7.84 MIN: 5.12 / MAX: 11.62 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: resnet50 OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: resnet50 b a c 4 8 12 16 20 SE +/- 0.10, N = 2 SE +/- 0.08, N = 2 SE +/- 0.14, N = 2 15.34 15.49 15.54 MIN: 15.07 / MAX: 21.69 MIN: 15.24 / MAX: 21.82 MIN: 15.15 / MAX: 27.2 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: yolov4-tiny OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: yolov4-tiny b a c 5 10 15 20 25 SE +/- 0.08, N = 2 SE +/- 0.02, N = 2 SE +/- 0.13, N = 2 20.51 20.66 20.80 MIN: 19.87 / MAX: 24.86 MIN: 20.04 / MAX: 25.04 MIN: 20.01 / MAX: 96.45 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: squeezenet_ssd OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: squeezenet_ssd b a c 4 8 12 16 20 SE +/- 0.09, N = 2 SE +/- 0.01, N = 2 SE +/- 0.62, N = 2 14.11 14.17 14.59 MIN: 13.32 / MAX: 18.63 MIN: 13.53 / MAX: 18.45 MIN: 13.37 / MAX: 277.61 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: regnety_400m OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: regnety_400m b c a 8 16 24 32 40 SE +/- 0.15, N = 2 SE +/- 0.07, N = 2 SE +/- 5.99, N = 2 27.54 27.59 35.24 MIN: 26.96 / MAX: 33.56 MIN: 26.64 / MAX: 33.41 MIN: 27.86 / MAX: 47.9 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: vision_transformer OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: vision_transformer c b a 11 22 33 44 55 SE +/- 0.30, N = 2 SE +/- 0.04, N = 2 SE +/- 0.07, N = 2 48.43 48.49 48.79 MIN: 47.33 / MAX: 85.35 MIN: 47.44 / MAX: 58.53 MIN: 47.65 / MAX: 78.36 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: FastestDet OpenBenchmarking.org ms, Fewer Is Better NCNN 20230517 Target: CPU - Model: FastestDet c b a 3 6 9 12 15 SE +/- 0.01, N = 2 SE +/- 0.11, N = 2 SE +/- 0.94, N = 2 8.88 9.04 10.25 MIN: 8.58 / MAX: 13.34 MIN: 8.65 / MAX: 15.19 MIN: 8.95 / MAX: 17.14 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream c a b 200 400 600 800 1000 SE +/- 0.59, N = 2 SE +/- 1.02, N = 2 SE +/- 0.16, N = 2 840.12 840.54 840.95
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream b c a 11 22 33 44 55 SE +/- 0.11, N = 2 SE +/- 0.05, N = 2 SE +/- 0.02, N = 2 49.92 49.93 50.13
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c 7 14 21 28 35 SE +/- 0.03, N = 2 SE +/- 0.03, N = 2 SE +/- 0.01, N = 2 28.92 28.95 28.96
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream a b c 1.3071 2.6142 3.9213 5.2284 6.5355 SE +/- 0.0325, N = 2 SE +/- 0.0212, N = 2 SE +/- 0.0271, N = 2 5.7470 5.7863 5.8092
Neural Magic DeepSparse Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Stream a b c 15 30 45 60 75 SE +/- 0.10, N = 2 SE +/- 0.10, N = 2 SE +/- 0.99, N = 2 65.28 65.74 66.28
Neural Magic DeepSparse Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Stream a b c 3 6 9 12 15 SE +/- 0.08, N = 2 SE +/- 0.01, N = 2 SE +/- 0.06, N = 2 11.54 11.57 11.58
Neural Magic DeepSparse Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream c b a 50 100 150 200 250 SE +/- 0.05, N = 2 SE +/- 0.26, N = 2 SE +/- 0.03, N = 2 222.82 223.37 223.47
Neural Magic DeepSparse Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream b c a 6 12 18 24 30 SE +/- 0.07, N = 2 SE +/- 0.01, N = 2 SE +/- 0.04, N = 2 25.27 25.30 25.36
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream c b a 15 30 45 60 75 SE +/- 0.01, N = 2 SE +/- 0.04, N = 2 SE +/- 0.04, N = 2 68.25 68.28 68.29
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream c a b 2 4 6 8 10 SE +/- 0.0308, N = 2 SE +/- 0.0166, N = 2 SE +/- 0.0087, N = 2 6.2238 6.2525 6.2566
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream b c a 2 4 6 8 10 SE +/- 0.0212, N = 2 SE +/- 0.0366, N = 2 SE +/- 0.0013, N = 2 8.3441 8.3468 8.3665
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream b c a 0.3101 0.6202 0.9303 1.2404 1.5505 SE +/- 0.0055, N = 2 SE +/- 0.0010, N = 2 SE +/- 0.0205, N = 2 1.3623 1.3634 1.3784
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream c b a 30 60 90 120 150 SE +/- 0.05, N = 2 SE +/- 0.09, N = 2 SE +/- 0.13, N = 2 141.48 141.62 141.70
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream c b a 2 4 6 8 10 SE +/- 0.0003, N = 2 SE +/- 0.0012, N = 2 SE +/- 0.0056, N = 2 8.3300 8.3368 8.3423
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream c b a 150 300 450 600 750 SE +/- 0.34, N = 2 SE +/- 0.17, N = 2 SE +/- 0.37, N = 2 679.81 679.82 681.28
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream b c a 9 18 27 36 45 SE +/- 0.01, N = 2 SE +/- 0.01, N = 2 SE +/- 0.01, N = 2 40.70 40.76 40.90
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream b a c 15 30 45 60 75 SE +/- 0.13, N = 2 SE +/- 0.04, N = 2 SE +/- 0.01, N = 2 68.27 68.32 68.34
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream c a b 2 4 6 8 10 SE +/- 0.0244, N = 2 SE +/- 0.0097, N = 2 SE +/- 0.0121, N = 2 6.2195 6.2481 6.2495
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream b c a 30 60 90 120 150 SE +/- 0.12, N = 2 SE +/- 0.23, N = 2 SE +/- 0.07, N = 2 140.26 140.30 140.40
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream c a b 2 4 6 8 10 SE +/- 0.0053, N = 2 SE +/- 0.0127, N = 2 SE +/- 0.0014, N = 2 8.2903 8.3056 8.3182
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream b c a 20 40 60 80 100 SE +/- 0.15, N = 2 SE +/- 0.16, N = 2 SE +/- 0.00, N = 2 97.84 97.87 97.87
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream b a c 3 6 9 12 15 SE +/- 0.02, N = 2 SE +/- 0.00, N = 2 SE +/- 0.01, N = 2 10.24 10.26 10.31
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream c b a 130 260 390 520 650 SE +/- 0.01, N = 2 SE +/- 0.12, N = 2 SE +/- 0.05, N = 2 596.53 596.81 597.97
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream c b a 8 16 24 32 40 SE +/- 0.03, N = 2 SE +/- 0.05, N = 2 SE +/- 0.03, N = 2 34.90 34.91 34.91
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream b a c 12 24 36 48 60 SE +/- 0.06, N = 2 SE +/- 0.03, N = 2 SE +/- 0.04, N = 2 55.56 55.58 55.58
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream c a b 3 6 9 12 15 SE +/- 0.02, N = 2 SE +/- 0.01, N = 2 SE +/- 0.07, N = 2 10.57 10.58 10.63
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream b c a 40 80 120 160 200 SE +/- 0.00, N = 2 SE +/- 0.21, N = 2 SE +/- 0.06, N = 2 192.16 192.21 192.35
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream c a b 5 10 15 20 25 SE +/- 0.03, N = 2 SE +/- 0.03, N = 2 SE +/- 0.00, N = 2 18.53 18.59 18.61
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream b c a 200 400 600 800 1000 SE +/- 0.01, N = 2 SE +/- 0.44, N = 2 SE +/- 0.49, N = 2 840.26 840.44 841.48
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.5 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream a c b 11 22 33 44 55 SE +/- 0.05, N = 2 SE +/- 0.06, N = 2 SE +/- 0.00, N = 2 49.81 49.86 49.91
Timed GCC Compilation Time To Compile OpenBenchmarking.org Seconds, Fewer Is Better Timed GCC Compilation 13.2 Time To Compile a c b 200 400 600 800 1000 SE +/- 1.81, N = 2 SE +/- 0.66, N = 2 SE +/- 0.03, N = 2 1020.13 1020.22 1020.85
Apache CouchDB Bulk Size: 100 - Inserts: 1000 - Rounds: 30 OpenBenchmarking.org Seconds, Fewer Is Better Apache CouchDB 3.3.2 Bulk Size: 100 - Inserts: 1000 - Rounds: 30 a 20 40 60 80 100 SE +/- 0.50, N = 2 101.58 1. (CXX) g++ options: -std=c++17 -lmozjs-78 -lm -lei -fPIC -MMD
Apache CouchDB Bulk Size: 100 - Inserts: 3000 - Rounds: 30 OpenBenchmarking.org Seconds, Fewer Is Better Apache CouchDB 3.3.2 Bulk Size: 100 - Inserts: 3000 - Rounds: 30 a 80 160 240 320 400 SE +/- 0.25, N = 2 346.09 1. (CXX) g++ options: -std=c++17 -lmozjs-78 -lm -lei -fPIC -MMD
Apache CouchDB Bulk Size: 300 - Inserts: 1000 - Rounds: 30 OpenBenchmarking.org Seconds, Fewer Is Better Apache CouchDB 3.3.2 Bulk Size: 300 - Inserts: 1000 - Rounds: 30 a 40 80 120 160 200 SE +/- 0.64, N = 2 169.51 1. (CXX) g++ options: -std=c++17 -lmozjs-78 -lm -lei -fPIC -MMD
Apache CouchDB Bulk Size: 300 - Inserts: 3000 - Rounds: 30 OpenBenchmarking.org Seconds, Fewer Is Better Apache CouchDB 3.3.2 Bulk Size: 300 - Inserts: 3000 - Rounds: 30 a 120 240 360 480 600 SE +/- 0.52, N = 2 572.13 1. (CXX) g++ options: -std=c++17 -lmozjs-78 -lm -lei -fPIC -MMD
Apache CouchDB Bulk Size: 500 - Inserts: 1000 - Rounds: 30 OpenBenchmarking.org Seconds, Fewer Is Better Apache CouchDB 3.3.2 Bulk Size: 500 - Inserts: 1000 - Rounds: 30 a 70 140 210 280 350 SE +/- 8.78, N = 2 339.97 1. (CXX) g++ options: -std=c++17 -lmozjs-78 -lm -lei -fPIC -MMD
Apache CouchDB Bulk Size: 500 - Inserts: 3000 - Rounds: 30 OpenBenchmarking.org Seconds, Fewer Is Better Apache CouchDB 3.3.2 Bulk Size: 500 - Inserts: 3000 - Rounds: 30 a 500 1000 1500 2000 2500 2390.93 1. (CXX) g++ options: -std=c++17 -lmozjs-78 -lm -lei -fPIC -MMD
Blender Blend File: BMW27 - Compute: CPU-Only OpenBenchmarking.org Seconds, Fewer Is Better Blender 3.6 Blend File: BMW27 - Compute: CPU-Only c a b 6 12 18 24 30 SE +/- 0.04, N = 2 SE +/- 0.06, N = 2 SE +/- 0.15, N = 2 27.24 27.27 27.50
Blender Blend File: Classroom - Compute: CPU-Only OpenBenchmarking.org Seconds, Fewer Is Better Blender 3.6 Blend File: Classroom - Compute: CPU-Only b c a 15 30 45 60 75 SE +/- 0.03, N = 2 SE +/- 0.13, N = 2 SE +/- 0.14, N = 2 68.50 68.70 68.80
Blender Blend File: Fishy Cat - Compute: CPU-Only OpenBenchmarking.org Seconds, Fewer Is Better Blender 3.6 Blend File: Fishy Cat - Compute: CPU-Only a c b 8 16 24 32 40 SE +/- 0.02, N = 2 SE +/- 0.01, N = 2 SE +/- 0.25, N = 2 33.70 33.72 33.76
Blender Blend File: Barbershop - Compute: CPU-Only OpenBenchmarking.org Seconds, Fewer Is Better Blender 3.6 Blend File: Barbershop - Compute: CPU-Only c a b 60 120 180 240 300 SE +/- 0.52, N = 2 SE +/- 0.11, N = 2 SE +/- 0.23, N = 2 253.43 253.49 253.77
Blender Blend File: Pabellon Barcelona - Compute: CPU-Only OpenBenchmarking.org Seconds, Fewer Is Better Blender 3.6 Blend File: Pabellon Barcelona - Compute: CPU-Only c b a 20 40 60 80 100 SE +/- 0.04, N = 2 SE +/- 0.14, N = 2 SE +/- 0.40, N = 2 84.17 84.35 84.55
Phoronix Test Suite v10.8.4