7763 2204

AMD EPYC 7763 64-Core testing with a AMD DAYTONA_X (RYM1009B BIOS) and ASPEED on Ubuntu 22.04 via the Phoronix Test Suite.

HTML result view exported from: https://openbenchmarking.org/result/2308059-NE-77632204529&grs&sor.

7763 2204ProcessorMotherboardChipsetMemoryDiskGraphicsMonitorNetworkOSKernelDesktopDisplay ServerVulkanCompilerFile-SystemScreen ResolutionabcAMD EPYC 7763 64-Core @ 2.45GHz (64 Cores / 128 Threads)AMD DAYTONA_X (RYM1009B BIOS)AMD Starship/Matisse256GB800GB INTEL SSDPF21Q800GBASPEEDVE2282 x Mellanox MT27710Ubuntu 22.046.2.0-phx (x86_64)GNOME Shell 42.5X Server 1.21.1.31.3.224GCC 11.3.0 + LLVM 14.0.0ext41920x1080OpenBenchmarking.orgKernel Details- Transparent Huge Pages: madviseCompiler Details- --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details- Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa001173 Java Details- OpenJDK Runtime Environment (build 11.0.20+8-post-Ubuntu-1ubuntu122.04)Python Details- Python 3.10.6Security Details- itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected

7763 2204ncnn: CPU - regnety_400mapache-iotdb: 500 - 1 - 500ncnn: CPU - shufflenet-v2apache-iotdb: 200 - 1 - 200apache-iotdb: 500 - 1 - 500apache-iotdb: 500 - 1 - 200ncnn: CPU - blazefaceapache-iotdb: 500 - 1 - 200ncnn: CPU - FastestDetapache-iotdb: 200 - 1 - 200ncnn: CPU-v3-v3 - mobilenet-v3apache-iotdb: 100 - 1 - 200srsran: Downlink Processor Benchmarkapache-iotdb: 500 - 100 - 200apache-iotdb: 500 - 100 - 500apache-iotdb: 100 - 100 - 200apache-iotdb: 100 - 1 - 500apache-iotdb: 500 - 100 - 200apache-iotdb: 500 - 100 - 500apache-iotdb: 100 - 100 - 200ncnn: CPU - mnasnetapache-iotdb: 100 - 100 - 500apache-iotdb: 200 - 100 - 500apache-iotdb: 100 - 100 - 500apache-iotdb: 100 - 1 - 200apache-iotdb: 200 - 100 - 200apache-iotdb: 200 - 1 - 500ncnn: CPU - squeezenet_ssdapache-iotdb: 200 - 100 - 500apache-iotdb: 100 - 1 - 500ncnn: CPU-v2-v2 - mobilenet-v2apache-iotdb: 200 - 1 - 500ncnn: CPU - efficientnet-b0apache-iotdb: 200 - 100 - 200vvenc: Bosphorus 4K - Fasterdeepsparse: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Asynchronous Multi-Streamdeepsparse: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Asynchronous Multi-Streamncnn: CPU - yolov4-tinycassandra: Writessrsran: PUSCH Processor Benchmark, Throughput Threadncnn: CPU - resnet50deepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Streamdeepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Streamncnn: CPU - vgg16deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Streamdeepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Streamncnn: CPU - resnet18ncnn: CPU - googlenetncnn: CPU - mobilenetblender: BMW27 - CPU-Onlydeepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Streamdeepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Streamncnn: CPU - vision_transformerdeepsparse: BERT-Large, NLP Question Answering - Asynchronous Multi-Streambrl-cad: VGR Performance Metricdeepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Streamdeepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Streamdeepsparse: ResNet-50, Baseline - Synchronous Single-Streamdeepsparse: ResNet-50, Baseline - Synchronous Single-Streamdeepsparse: BERT-Large, NLP Question Answering - Synchronous Single-Streamdeepsparse: BERT-Large, NLP Question Answering - Synchronous Single-Streamdeepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Streamdeepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Streamdeepsparse: NLP Text Classification, BERT base uncased SST2 - Synchronous Single-Streamdeepsparse: NLP Text Classification, BERT base uncased SST2 - Synchronous Single-Streamsrsran: PUSCH Processor Benchmark, Throughput Totalblender: Pabellon Barcelona - CPU-Onlyblender: Classroom - CPU-Onlydeepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Streamdeepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Streamvvenc: Bosphorus 1080p - Fasterdeepsparse: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Synchronous Single-Streamdeepsparse: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Synchronous Single-Streamdeepsparse: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Asynchronous Multi-Streamdeepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Streamdeepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Streamdeepsparse: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Synchronous Single-Streamdeepsparse: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Synchronous Single-Streamdeepsparse: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Asynchronous Multi-Streamvvenc: Bosphorus 4K - Fastdeepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Streamdeepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Streamdeepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Streamdeepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Streamdeepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Streamvvenc: Bosphorus 1080p - Fastdeepsparse: CV Detection, YOLOv5s COCO - Asynchronous Multi-Streamdeepsparse: BERT-Large, NLP Question Answering - Asynchronous Multi-Streamdeepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Streamdeepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Streamdeepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Streamncnn: CPU - alexnetdeepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Streamblender: Fishy Cat - CPU-Onlydeepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Streamdeepsparse: CV Detection, YOLOv5s COCO - Asynchronous Multi-Streamdeepsparse: CV Detection, YOLOv5s COCO - Synchronous Single-Streamdeepsparse: CV Detection, YOLOv5s COCO - Synchronous Single-Streamdeepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Streamdeepsparse: NLP Text Classification, BERT base uncased SST2 - Asynchronous Multi-Streamblender: Barbershop - CPU-Onlydeepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Streamdeepsparse: NLP Text Classification, BERT base uncased SST2 - Asynchronous Multi-Streamdeepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Streamdeepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Streamdeepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Streamdeepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Streamdeepsparse: ResNet-50, Baseline - Asynchronous Multi-Streamdeepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Streambuild-gcc: Time To Compiledeepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Streamdeepsparse: ResNet-50, Baseline - Asynchronous Multi-Streamdeepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Streamdeepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Streamdeepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Streamdeepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Streamcouchdb: 500 - 3000 - 30couchdb: 500 - 1000 - 30couchdb: 300 - 3000 - 30couchdb: 300 - 1000 - 30couchdb: 100 - 3000 - 30couchdb: 100 - 1000 - 30abc35.2427.19.0915.241636128.7313.543.971182440.6210.25898967.087.0017.45657.735.0556935634.5536.0434.3651341708.8581.8139287432.926.0981.2109.3851316464.44644019.7235.0933.514.1742048733.221038515.626.351232509.199.9846437377.6710.646489.770965.275020.66236650211.115.491.3784723.714923.84173.94625.7478.5014.6214.1127.2797.356110.264548.7946.720473438610.577094.5097159.84926.252540.904824.4422159.96606.248118.585853.78459682.184.5568.8050.125819.946929.35239.421925.356143.0725120.36458.305611.537886.6104223.47155.9918.36653814.519453.4831597.967937.614416.083225.3104681.275649.813220.072037.61095.231105.379133.7028.9150141.70368.3423119.8041841.4779165.9980253.49467.9848192.354668.3188840.5416227.4188140.3979468.1383326.31231020.133574.955068.285255.581728.631034.910497.87112390.933339.967572.125169.505346.085101.57827.54267.9313.61686943.1611.633.431365831.59.04978176.766.5517.28658.136.159505306.5536.8532.9250045888.9879.8338401769.175.982.26110.8850507747.12648308.2733.8433.8614.1141987111.391069145.796.261226219.889.7847245476.7810.815486.172865.735920.51238161208.215.341.3623732.110123.64172.75055.78638.4214.5313.9727.597.612110.238048.4946.573972987610.629994.0445159.74816.256640.699324.5655159.92386.249518.614453.70239718.684.3568.5049.922620.027929.3939.563125.2659143.2058120.17628.318211.566686.3905223.36705.9938.34413824.304553.5531596.809137.531516.090225.4592679.824049.914420.031537.53525.221104.037733.7628.9451141.61858.3368119.8818840.2573166.2231253.77468.1087192.164568.2716840.9493227.6423140.2607467.9696326.54391020.846575.285968.276355.560128.632934.908297.843427.5931.47.6016.071446487.711.833.481367763.498.88870795.926.3416.35619.337.1256463717.5435.0134.0849201448.8183.1439945212.995.8679.16106.7352464142.83667880.9634.7932.7114.5943363203.761044153.446.171261385.899.7546674344.6910.818482.127466.283720.80234887210.815.541.3634731.509923.91172.06595.80928.5114.4714.0327.2496.882210.314648.4346.902173043410.572094.5548160.59326.223840.760624.5284160.69856.219518.52853.9529727.184.1768.7049.931220.024529.46939.511725.2989143.5714120.58138.290311.576486.3219222.81755.9768.34683823.083353.6166596.534337.581416.055225.8007679.811849.858020.053637.57555.221103.355233.7228.9644141.47758.3300119.9790840.4350166.058253.43467.6396192.214868.3388840.1234227.5740140.3036468.3293326.41081020.216575.115868.250355.583528.641634.897897.8698OpenBenchmarking.org

NCNN

Target: CPU - Model: regnety_400m

OpenBenchmarking.orgms, Fewer Is BetterNCNN 20230517Target: CPU - Model: regnety_400mbca816243240SE +/- 0.15, N = 2SE +/- 0.07, N = 2SE +/- 5.99, N = 227.5427.5935.24MIN: 26.96 / MAX: 33.56MIN: 26.64 / MAX: 33.41MIN: 27.86 / MAX: 47.91. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread

Apache IoTDB

Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 500

OpenBenchmarking.orgAverage Latency, More Is BetterApache IoTDB 1.1.2Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 500cab71421283531.427.126.0MAX: 890.05MAX: 934.45MAX: 873.88

NCNN

Target: CPU - Model: shufflenet-v2

OpenBenchmarking.orgms, Fewer Is BetterNCNN 20230517Target: CPU - Model: shufflenet-v2cba3691215SE +/- 0.04, N = 2SE +/- 0.31, N = 2SE +/- 1.23, N = 27.607.939.09MIN: 7.44 / MAX: 11.6MIN: 7.51 / MAX: 11.45MIN: 7.74 / MAX: 15.921. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread

Apache IoTDB

Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 200

OpenBenchmarking.orgAverage Latency, More Is BetterApache IoTDB 1.1.2Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 200cab4812162016.0715.2413.60MAX: 592.48MAX: 583.94MAX: 586.94

Apache IoTDB

Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 500

OpenBenchmarking.orgpoint/sec, More Is BetterApache IoTDB 1.1.2Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 500bac400K800K1200K1600K2000K1686943.161636128.731446487.70

Apache IoTDB

Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 200

OpenBenchmarking.orgAverage Latency, More Is BetterApache IoTDB 1.1.2Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 200acb369121513.5411.8311.63MAX: 856.65MAX: 836.9MAX: 860.78

NCNN

Target: CPU - Model: blazeface

OpenBenchmarking.orgms, Fewer Is BetterNCNN 20230517Target: CPU - Model: blazefacebca0.89331.78662.67993.57324.4665SE +/- 0.01, N = 2SE +/- 0.05, N = 2SE +/- 0.09, N = 23.433.483.97MIN: 3.35 / MAX: 3.83MIN: 3.32 / MAX: 8.58MIN: 3.5 / MAX: 7.611. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread

Apache IoTDB

Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 200

OpenBenchmarking.orgpoint/sec, More Is BetterApache IoTDB 1.1.2Device Count: 500 - Batch Size Per Write: 1 - Sensor Count: 200cba300K600K900K1200K1500K1367763.491365831.501182440.62

NCNN

Target: CPU - Model: FastestDet

OpenBenchmarking.orgms, Fewer Is BetterNCNN 20230517Target: CPU - Model: FastestDetcba3691215SE +/- 0.01, N = 2SE +/- 0.11, N = 2SE +/- 0.94, N = 28.889.0410.25MIN: 8.58 / MAX: 13.34MIN: 8.65 / MAX: 15.19MIN: 8.95 / MAX: 17.141. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread

Apache IoTDB

Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 200

OpenBenchmarking.orgpoint/sec, More Is BetterApache IoTDB 1.1.2Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 200bac200K400K600K800K1000K978176.76898967.08870795.92

NCNN

Target: CPU-v3-v3 - Model: mobilenet-v3

OpenBenchmarking.orgms, Fewer Is BetterNCNN 20230517Target: CPU-v3-v3 - Model: mobilenet-v3cba246810SE +/- 0.04, N = 2SE +/- 0.18, N = 2SE +/- 0.55, N = 26.346.557.00MIN: 6.15 / MAX: 11.57MIN: 6.24 / MAX: 7.61MIN: 6.3 / MAX: 10.21. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread

Apache IoTDB

Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 200

OpenBenchmarking.orgAverage Latency, More Is BetterApache IoTDB 1.1.2Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 200abc4812162017.4517.2816.35MAX: 645.35MAX: 644.33MAX: 668.86

srsRAN Project

Test: Downlink Processor Benchmark

OpenBenchmarking.orgMbps, More Is BettersrsRAN Project 23.5Test: Downlink Processor Benchmarkbac140280420560700SE +/- 27.75, N = 2SE +/- 17.85, N = 2SE +/- 0.35, N = 2658.1657.7619.31. (CXX) g++ options: -march=native -mfma -O3 -fno-trapping-math -fno-math-errno -lgtest

Apache IoTDB

Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 200

OpenBenchmarking.orgAverage Latency, More Is BetterApache IoTDB 1.1.2Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 200cba91827364537.1236.1035.05MAX: 2182.81MAX: 1990.15MAX: 2157.23

Apache IoTDB

Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 500

OpenBenchmarking.orgpoint/sec, More Is BetterApache IoTDB 1.1.2Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 500bac13M26M39M52M65M59505306.5556935634.5556463717.54

Apache IoTDB

Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 200

OpenBenchmarking.orgAverage Latency, More Is BetterApache IoTDB 1.1.2Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 200bac81624324036.8536.0435.01MAX: 721.27MAX: 804.01MAX: 746.4

Apache IoTDB

Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 500

OpenBenchmarking.orgAverage Latency, More Is BetterApache IoTDB 1.1.2Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 500acb81624324034.3634.0832.92MAX: 704.53MAX: 699.28MAX: 728.63

Apache IoTDB

Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 200

OpenBenchmarking.orgpoint/sec, More Is BetterApache IoTDB 1.1.2Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 200abc11M22M33M44M55M51341708.8550045888.9849201448.81

Apache IoTDB

Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 500

OpenBenchmarking.orgAverage Latency, More Is BetterApache IoTDB 1.1.2Device Count: 500 - Batch Size Per Write: 100 - Sensor Count: 500cab2040608010083.1481.8179.83MAX: 2932.1MAX: 3018.16MAX: 1607.86

Apache IoTDB

Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 200

OpenBenchmarking.orgpoint/sec, More Is BetterApache IoTDB 1.1.2Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 200cab9M18M27M36M45M39945212.9939287432.9238401769.17

NCNN

Target: CPU - Model: mnasnet

OpenBenchmarking.orgms, Fewer Is BetterNCNN 20230517Target: CPU - Model: mnasnetcba246810SE +/- 0.05, N = 2SE +/- 0.00, N = 2SE +/- 0.11, N = 25.865.906.09MIN: 5.73 / MAX: 11.67MIN: 5.81 / MAX: 12.33MIN: 5.89 / MAX: 10.351. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread

Apache IoTDB

Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 500

OpenBenchmarking.orgAverage Latency, More Is BetterApache IoTDB 1.1.2Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 500bac2040608010082.2681.2079.16MAX: 864.29MAX: 1009.28MAX: 1006.03

Apache IoTDB

Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 500

OpenBenchmarking.orgAverage Latency, More Is BetterApache IoTDB 1.1.2Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 500bac20406080100110.88109.38106.73MAX: 3569.78MAX: 3597.09MAX: 3485.91

Apache IoTDB

Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 500

OpenBenchmarking.orgpoint/sec, More Is BetterApache IoTDB 1.1.2Device Count: 100 - Batch Size Per Write: 100 - Sensor Count: 500cab11M22M33M44M55M52464142.8351316464.4450507747.12

Apache IoTDB

Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 200

OpenBenchmarking.orgpoint/sec, More Is BetterApache IoTDB 1.1.2Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 200cba140K280K420K560K700K667880.96648308.27644019.72

Apache IoTDB

Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 200

OpenBenchmarking.orgAverage Latency, More Is BetterApache IoTDB 1.1.2Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 200acb81624324035.0934.7933.84MAX: 804.64MAX: 780.01MAX: 773.52

Apache IoTDB

Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 500

OpenBenchmarking.orgAverage Latency, More Is BetterApache IoTDB 1.1.2Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 500bac81624324033.8633.5032.71MAX: 659.59MAX: 690.29MAX: 725.08

NCNN

Target: CPU - Model: squeezenet_ssd

OpenBenchmarking.orgms, Fewer Is BetterNCNN 20230517Target: CPU - Model: squeezenet_ssdbac48121620SE +/- 0.09, N = 2SE +/- 0.01, N = 2SE +/- 0.62, N = 214.1114.1714.59MIN: 13.32 / MAX: 18.63MIN: 13.53 / MAX: 18.45MIN: 13.37 / MAX: 277.611. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread

Apache IoTDB

Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 500

OpenBenchmarking.orgpoint/sec, More Is BetterApache IoTDB 1.1.2Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 500cab9M18M27M36M45M43363203.7642048733.2241987111.39

Apache IoTDB

Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 500

OpenBenchmarking.orgpoint/sec, More Is BetterApache IoTDB 1.1.2Device Count: 100 - Batch Size Per Write: 1 - Sensor Count: 500bca200K400K600K800K1000K1069145.791044153.441038515.62

NCNN

Target: CPU-v2-v2 - Model: mobilenet-v2

OpenBenchmarking.orgms, Fewer Is BetterNCNN 20230517Target: CPU-v2-v2 - Model: mobilenet-v2cba246810SE +/- 0.07, N = 2SE +/- 0.00, N = 2SE +/- 0.01, N = 26.176.266.35MIN: 6.02 / MAX: 6.83MIN: 6.11 / MAX: 12.76MIN: 6.19 / MAX: 12.451. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread

Apache IoTDB

Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 500

OpenBenchmarking.orgpoint/sec, More Is BetterApache IoTDB 1.1.2Device Count: 200 - Batch Size Per Write: 1 - Sensor Count: 500cab300K600K900K1200K1500K1261385.891232509.191226219.88

NCNN

Target: CPU - Model: efficientnet-b0

OpenBenchmarking.orgms, Fewer Is BetterNCNN 20230517Target: CPU - Model: efficientnet-b0cba3691215SE +/- 0.04, N = 2SE +/- 0.01, N = 2SE +/- 0.03, N = 29.759.789.98MIN: 9.58 / MAX: 13.38MIN: 9.64 / MAX: 16.01MIN: 9.82 / MAX: 10.961. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread

Apache IoTDB

Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 200

OpenBenchmarking.orgpoint/sec, More Is BetterApache IoTDB 1.1.2Device Count: 200 - Batch Size Per Write: 100 - Sensor Count: 200bca10M20M30M40M50M47245476.7846674344.6946437377.67

VVenC

Video Input: Bosphorus 4K - Video Preset: Faster

OpenBenchmarking.orgFrames Per Second, More Is BetterVVenC 1.9Video Input: Bosphorus 4K - Video Preset: Fastercba3691215SE +/- 0.00, N = 2SE +/- 0.02, N = 2SE +/- 0.18, N = 210.8210.8210.651. (CXX) g++ options: -O3 -flto -fno-fat-lto-objects -flto=auto

Neural Magic DeepSparse

Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Streamabc110220330440550SE +/- 0.68, N = 2SE +/- 1.03, N = 2SE +/- 7.21, N = 2489.77486.17482.13

Neural Magic DeepSparse

Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Streamabc1530456075SE +/- 0.10, N = 2SE +/- 0.10, N = 2SE +/- 0.99, N = 265.2865.7466.28

NCNN

Target: CPU - Model: yolov4-tiny

OpenBenchmarking.orgms, Fewer Is BetterNCNN 20230517Target: CPU - Model: yolov4-tinybac510152025SE +/- 0.08, N = 2SE +/- 0.02, N = 2SE +/- 0.13, N = 220.5120.6620.80MIN: 19.87 / MAX: 24.86MIN: 20.04 / MAX: 25.04MIN: 20.01 / MAX: 96.451. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread

Apache Cassandra

Test: Writes

OpenBenchmarking.orgOp/s, More Is BetterApache Cassandra 4.1.3Test: Writesbac50K100K150K200K250KSE +/- 817.50, N = 2SE +/- 633.50, N = 2SE +/- 669.00, N = 2238161236650234887

srsRAN Project

Test: PUSCH Processor Benchmark, Throughput Thread

OpenBenchmarking.orgMbps, More Is BettersrsRAN Project 23.5Test: PUSCH Processor Benchmark, Throughput Threadacb50100150200250SE +/- 0.10, N = 2SE +/- 0.20, N = 2SE +/- 1.90, N = 2211.1210.8208.21. (CXX) g++ options: -march=native -mfma -O3 -fno-trapping-math -fno-math-errno -lgtest

NCNN

Target: CPU - Model: resnet50

OpenBenchmarking.orgms, Fewer Is BetterNCNN 20230517Target: CPU - Model: resnet50bac48121620SE +/- 0.10, N = 2SE +/- 0.08, N = 2SE +/- 0.14, N = 215.3415.4915.54MIN: 15.07 / MAX: 21.69MIN: 15.24 / MAX: 21.82MIN: 15.15 / MAX: 27.21. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread

Neural Magic DeepSparse

Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Streambca0.31010.62020.93031.24041.5505SE +/- 0.0055, N = 2SE +/- 0.0010, N = 2SE +/- 0.0205, N = 21.36231.36341.3784

Neural Magic DeepSparse

Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Streambca160320480640800SE +/- 2.91, N = 2SE +/- 0.48, N = 2SE +/- 10.71, N = 2732.11731.51723.71

NCNN

Target: CPU - Model: vgg16

OpenBenchmarking.orgms, Fewer Is BetterNCNN 20230517Target: CPU - Model: vgg16bac612182430SE +/- 0.06, N = 2SE +/- 0.05, N = 2SE +/- 0.08, N = 223.6423.8423.91MIN: 23.33 / MAX: 28.05MIN: 23.45 / MAX: 28.55MIN: 23.48 / MAX: 30.631. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread

Neural Magic DeepSparse

Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Streamabc4080120160200SE +/- 0.98, N = 2SE +/- 0.63, N = 2SE +/- 0.80, N = 2173.95172.75172.07

Neural Magic DeepSparse

Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Streamabc1.30712.61423.92135.22846.5355SE +/- 0.0325, N = 2SE +/- 0.0212, N = 2SE +/- 0.0271, N = 25.74705.78635.8092

NCNN

Target: CPU - Model: resnet18

OpenBenchmarking.orgms, Fewer Is BetterNCNN 20230517Target: CPU - Model: resnet18bac246810SE +/- 0.04, N = 2SE +/- 0.03, N = 2SE +/- 0.03, N = 28.428.508.51MIN: 8.27 / MAX: 14.6MIN: 8.33 / MAX: 14.69MIN: 8.3 / MAX: 13.611. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread

NCNN

Target: CPU - Model: googlenet

OpenBenchmarking.orgms, Fewer Is BetterNCNN 20230517Target: CPU - Model: googlenetcba48121620SE +/- 0.06, N = 2SE +/- 0.02, N = 2SE +/- 0.03, N = 214.4714.5314.62MIN: 14.26 / MAX: 20.57MIN: 14.31 / MAX: 24.11MIN: 14.46 / MAX: 25.511. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread

NCNN

Target: CPU - Model: mobilenet

OpenBenchmarking.orgms, Fewer Is BetterNCNN 20230517Target: CPU - Model: mobilenetbca48121620SE +/- 0.03, N = 2SE +/- 0.09, N = 2SE +/- 0.04, N = 213.9714.0314.11MIN: 13.64 / MAX: 19.68MIN: 13.68 / MAX: 18.14MIN: 13.76 / MAX: 19.751. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread

Blender

Blend File: BMW27 - Compute: CPU-Only

OpenBenchmarking.orgSeconds, Fewer Is BetterBlender 3.6Blend File: BMW27 - Compute: CPU-Onlycab612182430SE +/- 0.04, N = 2SE +/- 0.06, N = 2SE +/- 0.15, N = 227.2427.2727.50

Neural Magic DeepSparse

Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Streambac20406080100SE +/- 0.17, N = 2SE +/- 0.04, N = 2SE +/- 0.08, N = 297.6197.3696.88

Neural Magic DeepSparse

Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Streambac3691215SE +/- 0.02, N = 2SE +/- 0.00, N = 2SE +/- 0.01, N = 210.2410.2610.31

NCNN

Target: CPU - Model: vision_transformer

OpenBenchmarking.orgms, Fewer Is BetterNCNN 20230517Target: CPU - Model: vision_transformercba1122334455SE +/- 0.30, N = 2SE +/- 0.04, N = 2SE +/- 0.07, N = 248.4348.4948.79MIN: 47.33 / MAX: 85.35MIN: 47.44 / MAX: 58.53MIN: 47.65 / MAX: 78.361. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread

Neural Magic DeepSparse

Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Streamcab1122334455SE +/- 0.02, N = 2SE +/- 0.10, N = 2SE +/- 0.03, N = 246.9046.7246.57

BRL-CAD

VGR Performance Metric

OpenBenchmarking.orgVGR Performance Metric, More Is BetterBRL-CAD 7.36VGR Performance Metricacb160K320K480K640K800KSE +/- 1805.50, N = 2SE +/- 963.50, N = 2SE +/- 357.50, N = 27343867304347298761. (CXX) g++ options: -std=c++14 -pipe -fvisibility=hidden -fno-strict-aliasing -fno-common -fexceptions -ftemplate-depth-128 -m64 -ggdb3 -O3 -fipa-pta -fstrength-reduce -finline-functions -flto -ltcl8.6 -lregex_brl -lz_brl -lnetpbm -ldl -lm -ltk8.6

Neural Magic DeepSparse

Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Streamcab3691215SE +/- 0.02, N = 2SE +/- 0.01, N = 2SE +/- 0.07, N = 210.5710.5810.63

Neural Magic DeepSparse

Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Streamcab20406080100SE +/- 0.18, N = 2SE +/- 0.05, N = 2SE +/- 0.64, N = 294.5594.5194.04

Neural Magic DeepSparse

Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: ResNet-50, Baseline - Scenario: Synchronous Single-Streamcab4080120160200SE +/- 0.80, N = 2SE +/- 0.43, N = 2SE +/- 0.22, N = 2160.59159.85159.75

Neural Magic DeepSparse

Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: ResNet-50, Baseline - Scenario: Synchronous Single-Streamcab246810SE +/- 0.0308, N = 2SE +/- 0.0166, N = 2SE +/- 0.0087, N = 26.22386.25256.2566

Neural Magic DeepSparse

Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Streambca918273645SE +/- 0.01, N = 2SE +/- 0.01, N = 2SE +/- 0.01, N = 240.7040.7640.90

Neural Magic DeepSparse

Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Streambca612182430SE +/- 0.01, N = 2SE +/- 0.00, N = 2SE +/- 0.01, N = 224.5724.5324.44

Neural Magic DeepSparse

Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Streamcab4080120160200SE +/- 0.62, N = 2SE +/- 0.24, N = 2SE +/- 0.32, N = 2160.70159.97159.92

Neural Magic DeepSparse

Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Streamcab246810SE +/- 0.0244, N = 2SE +/- 0.0097, N = 2SE +/- 0.0121, N = 26.21956.24816.2495

Neural Magic DeepSparse

Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Streamcab510152025SE +/- 0.03, N = 2SE +/- 0.03, N = 2SE +/- 0.00, N = 218.5318.5918.61

Neural Magic DeepSparse

Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Streamcab1224364860SE +/- 0.10, N = 2SE +/- 0.08, N = 2SE +/- 0.01, N = 253.9553.7853.70

srsRAN Project

Test: PUSCH Processor Benchmark, Throughput Total

OpenBenchmarking.orgMbps, More Is BettersrsRAN Project 23.5Test: PUSCH Processor Benchmark, Throughput Totalcba2K4K6K8K10KSE +/- 56.45, N = 2SE +/- 44.35, N = 2SE +/- 13.30, N = 29727.19718.69682.11. (CXX) g++ options: -march=native -mfma -O3 -fno-trapping-math -fno-math-errno -lgtest

Blender

Blend File: Pabellon Barcelona - Compute: CPU-Only

OpenBenchmarking.orgSeconds, Fewer Is BetterBlender 3.6Blend File: Pabellon Barcelona - Compute: CPU-Onlycba20406080100SE +/- 0.04, N = 2SE +/- 0.14, N = 2SE +/- 0.40, N = 284.1784.3584.55

Blender

Blend File: Classroom - Compute: CPU-Only

OpenBenchmarking.orgSeconds, Fewer Is BetterBlender 3.6Blend File: Classroom - Compute: CPU-Onlybca1530456075SE +/- 0.03, N = 2SE +/- 0.13, N = 2SE +/- 0.14, N = 268.5068.7068.80

Neural Magic DeepSparse

Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Streambca1122334455SE +/- 0.11, N = 2SE +/- 0.05, N = 2SE +/- 0.02, N = 249.9249.9350.13

Neural Magic DeepSparse

Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Streambca510152025SE +/- 0.04, N = 2SE +/- 0.02, N = 2SE +/- 0.01, N = 220.0320.0219.95

VVenC

Video Input: Bosphorus 1080p - Video Preset: Faster

OpenBenchmarking.orgFrames Per Second, More Is BetterVVenC 1.9Video Input: Bosphorus 1080p - Video Preset: Fastercba714212835SE +/- 0.07, N = 2SE +/- 0.11, N = 2SE +/- 0.07, N = 229.4729.3929.351. (CXX) g++ options: -O3 -flto -fno-fat-lto-objects -flto=auto

Neural Magic DeepSparse

Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Streambca918273645SE +/- 0.11, N = 2SE +/- 0.01, N = 2SE +/- 0.06, N = 239.5639.5139.42

Neural Magic DeepSparse

Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Streambca612182430SE +/- 0.07, N = 2SE +/- 0.01, N = 2SE +/- 0.04, N = 225.2725.3025.36

Neural Magic DeepSparse

Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Streamcba306090120150SE +/- 0.03, N = 2SE +/- 0.18, N = 2SE +/- 0.01, N = 2143.57143.21143.07

Neural Magic DeepSparse

Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Streamcab306090120150SE +/- 0.08, N = 2SE +/- 0.19, N = 2SE +/- 0.02, N = 2120.58120.36120.18

Neural Magic DeepSparse

Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Streamcab246810SE +/- 0.0053, N = 2SE +/- 0.0127, N = 2SE +/- 0.0014, N = 28.29038.30568.3182

Neural Magic DeepSparse

Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Streamabc3691215SE +/- 0.08, N = 2SE +/- 0.01, N = 2SE +/- 0.06, N = 211.5411.5711.58

Neural Magic DeepSparse

Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Streamabc20406080100SE +/- 0.59, N = 2SE +/- 0.08, N = 2SE +/- 0.42, N = 286.6186.3986.32

Neural Magic DeepSparse

Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Streamcba50100150200250SE +/- 0.05, N = 2SE +/- 0.26, N = 2SE +/- 0.03, N = 2222.82223.37223.47

VVenC

Video Input: Bosphorus 4K - Video Preset: Fast

OpenBenchmarking.orgFrames Per Second, More Is BetterVVenC 1.9Video Input: Bosphorus 4K - Video Preset: Fastbac1.34842.69684.04525.39366.742SE +/- 0.006, N = 2SE +/- 0.001, N = 2SE +/- 0.002, N = 25.9935.9915.9761. (CXX) g++ options: -O3 -flto -fno-fat-lto-objects -flto=auto

Neural Magic DeepSparse

Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Streambca246810SE +/- 0.0212, N = 2SE +/- 0.0366, N = 2SE +/- 0.0013, N = 28.34418.34688.3665

Neural Magic DeepSparse

Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Streambca8001600240032004000SE +/- 10.54, N = 2SE +/- 17.23, N = 2SE +/- 0.54, N = 23824.303823.083814.52

Neural Magic DeepSparse

Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Streamcba1224364860SE +/- 0.00, N = 2SE +/- 0.05, N = 2SE +/- 0.00, N = 253.6253.5553.48

Neural Magic DeepSparse

Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Streamcba130260390520650SE +/- 0.01, N = 2SE +/- 0.12, N = 2SE +/- 0.05, N = 2596.53596.81597.97

Neural Magic DeepSparse

Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Streamacb918273645SE +/- 0.09, N = 2SE +/- 0.02, N = 2SE +/- 0.01, N = 237.6137.5837.53

VVenC

Video Input: Bosphorus 1080p - Video Preset: Fast

OpenBenchmarking.orgFrames Per Second, More Is BetterVVenC 1.9Video Input: Bosphorus 1080p - Video Preset: Fastbac48121620SE +/- 0.02, N = 2SE +/- 0.02, N = 2SE +/- 0.02, N = 216.0916.0816.061. (CXX) g++ options: -O3 -flto -fno-fat-lto-objects -flto=auto

Neural Magic DeepSparse

Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Streamcba50100150200250SE +/- 0.22, N = 2SE +/- 0.10, N = 2SE +/- 0.21, N = 2225.80225.46225.31

Neural Magic DeepSparse

Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Streamcba150300450600750SE +/- 0.34, N = 2SE +/- 0.17, N = 2SE +/- 0.37, N = 2679.81679.82681.28

Neural Magic DeepSparse

Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Streamacb1122334455SE +/- 0.05, N = 2SE +/- 0.06, N = 2SE +/- 0.00, N = 249.8149.8649.91

Neural Magic DeepSparse

Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Streamacb510152025SE +/- 0.02, N = 2SE +/- 0.02, N = 2SE +/- 0.00, N = 220.0720.0520.03

Neural Magic DeepSparse

Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Streamacb918273645SE +/- 0.04, N = 2SE +/- 0.02, N = 2SE +/- 0.01, N = 237.6137.5837.54

NCNN

Target: CPU - Model: alexnet

OpenBenchmarking.orgms, Fewer Is BetterNCNN 20230517Target: CPU - Model: alexnetbca1.17682.35363.53044.70725.884SE +/- 0.02, N = 2SE +/- 0.02, N = 2SE +/- 0.01, N = 25.225.225.23MIN: 5.11 / MAX: 5.77MIN: 5.12 / MAX: 7.84MIN: 5.12 / MAX: 11.621. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread

Neural Magic DeepSparse

Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Streamabc2004006008001000SE +/- 1.07, N = 2SE +/- 1.15, N = 2SE +/- 0.21, N = 21105.381104.041103.36

Blender

Blend File: Fishy Cat - Compute: CPU-Only

OpenBenchmarking.orgSeconds, Fewer Is BetterBlender 3.6Blend File: Fishy Cat - Compute: CPU-Onlyacb816243240SE +/- 0.02, N = 2SE +/- 0.01, N = 2SE +/- 0.25, N = 233.7033.7233.76

Neural Magic DeepSparse

Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Streamabc714212835SE +/- 0.03, N = 2SE +/- 0.03, N = 2SE +/- 0.01, N = 228.9228.9528.96

Neural Magic DeepSparse

Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Streamcba306090120150SE +/- 0.05, N = 2SE +/- 0.09, N = 2SE +/- 0.13, N = 2141.48141.62141.70

Neural Magic DeepSparse

Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Streamcba246810SE +/- 0.0003, N = 2SE +/- 0.0012, N = 2SE +/- 0.0056, N = 28.33008.33688.3423

Neural Magic DeepSparse

Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Streamcba306090120150SE +/- 0.01, N = 2SE +/- 0.02, N = 2SE +/- 0.08, N = 2119.98119.88119.80

Neural Magic DeepSparse

Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Streambca2004006008001000SE +/- 0.01, N = 2SE +/- 0.44, N = 2SE +/- 0.49, N = 2840.26840.44841.48

Neural Magic DeepSparse

Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Streambca4080120160200SE +/- 0.06, N = 2SE +/- 0.08, N = 2SE +/- 0.05, N = 2166.22166.06166.00

Blender

Blend File: Barbershop - Compute: CPU-Only

OpenBenchmarking.orgSeconds, Fewer Is BetterBlender 3.6Blend File: Barbershop - Compute: CPU-Onlycab60120180240300SE +/- 0.52, N = 2SE +/- 0.11, N = 2SE +/- 0.23, N = 2253.43253.49253.77

Neural Magic DeepSparse

Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Streambac100200300400500SE +/- 1.23, N = 2SE +/- 0.36, N = 2SE +/- 0.26, N = 2468.11467.98467.64

Neural Magic DeepSparse

Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Streambca4080120160200SE +/- 0.00, N = 2SE +/- 0.21, N = 2SE +/- 0.06, N = 2192.16192.21192.35

Neural Magic DeepSparse

Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Streambac1530456075SE +/- 0.13, N = 2SE +/- 0.04, N = 2SE +/- 0.01, N = 268.2768.3268.34

Neural Magic DeepSparse

Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Streamcab2004006008001000SE +/- 0.59, N = 2SE +/- 1.02, N = 2SE +/- 0.16, N = 2840.12840.54840.95

Neural Magic DeepSparse

Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Streambca50100150200250SE +/- 0.19, N = 2SE +/- 0.22, N = 2SE +/- 0.11, N = 2227.64227.57227.42

Neural Magic DeepSparse

Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Streambca306090120150SE +/- 0.12, N = 2SE +/- 0.23, N = 2SE +/- 0.07, N = 2140.26140.30140.40

Neural Magic DeepSparse

Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Streamcab100200300400500SE +/- 0.09, N = 2SE +/- 0.19, N = 2SE +/- 0.62, N = 2468.33468.14467.97

Neural Magic DeepSparse

Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Streambca70140210280350SE +/- 0.48, N = 2SE +/- 0.49, N = 2SE +/- 0.12, N = 2326.54326.41326.31

Timed GCC Compilation

Time To Compile

OpenBenchmarking.orgSeconds, Fewer Is BetterTimed GCC Compilation 13.2Time To Compileacb2004006008001000SE +/- 1.81, N = 2SE +/- 0.66, N = 2SE +/- 0.03, N = 21020.131020.221020.85

Neural Magic DeepSparse

Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Streambca120240360480600SE +/- 0.68, N = 2SE +/- 0.31, N = 2SE +/- 0.03, N = 2575.29575.12574.96

Neural Magic DeepSparse

Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Streamcba1530456075SE +/- 0.01, N = 2SE +/- 0.04, N = 2SE +/- 0.04, N = 268.2568.2868.29

Neural Magic DeepSparse

Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Streambac1224364860SE +/- 0.06, N = 2SE +/- 0.03, N = 2SE +/- 0.04, N = 255.5655.5855.58

Neural Magic DeepSparse

Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgitems/sec, More Is BetterNeural Magic DeepSparse 1.5Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Streamcba714212835SE +/- 0.02, N = 2SE +/- 0.04, N = 2SE +/- 0.03, N = 228.6428.6328.63

Neural Magic DeepSparse

Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Streamcba816243240SE +/- 0.03, N = 2SE +/- 0.05, N = 2SE +/- 0.03, N = 234.9034.9134.91

Neural Magic DeepSparse

Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.orgms/batch, Fewer Is BetterNeural Magic DeepSparse 1.5Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Streambca20406080100SE +/- 0.15, N = 2SE +/- 0.16, N = 2SE +/- 0.00, N = 297.8497.8797.87

Apache CouchDB

Bulk Size: 500 - Inserts: 3000 - Rounds: 30

OpenBenchmarking.orgSeconds, Fewer Is BetterApache CouchDB 3.3.2Bulk Size: 500 - Inserts: 3000 - Rounds: 30a50010001500200025002390.931. (CXX) g++ options: -std=c++17 -lmozjs-78 -lm -lei -fPIC -MMD

Apache CouchDB

Bulk Size: 500 - Inserts: 1000 - Rounds: 30

OpenBenchmarking.orgSeconds, Fewer Is BetterApache CouchDB 3.3.2Bulk Size: 500 - Inserts: 1000 - Rounds: 30a70140210280350SE +/- 8.78, N = 2339.971. (CXX) g++ options: -std=c++17 -lmozjs-78 -lm -lei -fPIC -MMD

Apache CouchDB

Bulk Size: 300 - Inserts: 3000 - Rounds: 30

OpenBenchmarking.orgSeconds, Fewer Is BetterApache CouchDB 3.3.2Bulk Size: 300 - Inserts: 3000 - Rounds: 30a120240360480600SE +/- 0.52, N = 2572.131. (CXX) g++ options: -std=c++17 -lmozjs-78 -lm -lei -fPIC -MMD

Apache CouchDB

Bulk Size: 300 - Inserts: 1000 - Rounds: 30

OpenBenchmarking.orgSeconds, Fewer Is BetterApache CouchDB 3.3.2Bulk Size: 300 - Inserts: 1000 - Rounds: 30a4080120160200SE +/- 0.64, N = 2169.511. (CXX) g++ options: -std=c++17 -lmozjs-78 -lm -lei -fPIC -MMD

Apache CouchDB

Bulk Size: 100 - Inserts: 3000 - Rounds: 30

OpenBenchmarking.orgSeconds, Fewer Is BetterApache CouchDB 3.3.2Bulk Size: 100 - Inserts: 3000 - Rounds: 30a80160240320400SE +/- 0.25, N = 2346.091. (CXX) g++ options: -std=c++17 -lmozjs-78 -lm -lei -fPIC -MMD

Apache CouchDB

Bulk Size: 100 - Inserts: 1000 - Rounds: 30

OpenBenchmarking.orgSeconds, Fewer Is BetterApache CouchDB 3.3.2Bulk Size: 100 - Inserts: 1000 - Rounds: 30a20406080100SE +/- 0.50, N = 2101.581. (CXX) g++ options: -std=c++17 -lmozjs-78 -lm -lei -fPIC -MMD

Phoronix Test Suite v10.8.5