Neural Magic DeepSparse

This is a benchmark of Neural Magic's DeepSparse using its built-in deepsparse.benchmark utility and various models from their SparseZoo (https://sparsezoo.neuralmagic.com/).

To run this test with the Phoronix Test Suite, the basic command is: phoronix-test-suite benchmark deepsparse.

Project Site

neuralmagic.com

Source Repository

github.com

Test Created

13 October 2022

Test Maintainer

Michael Larabel 

Test Type

System

Average Install Time

7 Minutes, 35 Seconds

Average Run Time

1 Minute, 43 Seconds

Test Dependencies

Python

Supported Platforms


CV Classification, ResNet-50 ImageNet14.3%NLP Text Classification, DistilBERT mnli14.5%NLP Token Classification, BERT base uncased conll200314.0%CV Detection,YOLOv5s COCO14.3%NLP Document Classification, oBERT base uncased on IMDB14.0%NLP Text Classification, BERT base uncased SST214.5%NLP Question Answering, BERT base uncased SQuaD 12layer Pruned9014.5%Model Option PopularityOpenBenchmarking.org
Synchronous Single-Stream49.7%Asynchronous Multi-Stream50.3%Scenario Option PopularityOpenBenchmarking.org

Revision History

pts/deepsparse-1.0.1   [View Source]   Thu, 13 Oct 2022 13:47:39 GMT
Initial commit of DeepSparse benchmark.

Suites Using This Test

Machine Learning

HPC - High Performance Computing


Performance Metrics

Analyze Test Configuration:

Neural Magic DeepSparse 1.1

Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.org metrics for this test profile configuration based on 279 public results since 13 October 2022 with the latest data as of 26 November 2022.

Below is an overview of the generalized performance for components where there is sufficient statistically significant data based upon user-uploaded results. It is important to keep in mind particularly in the Linux/open-source space there can be vastly different OS configurations, with this overview intended to offer just general guidance as to the performance expectations.

Component
Percentile Rank
# Compatible Public Results
items/sec (Average)
93rd
12
414 +/- 14
88th
5
357 +/- 1
86th
8
344 +/- 16
Mid-Tier
75th
< 208
Median
50th
88
Low-Tier
25th
< 34
OpenBenchmarking.orgDistribution Of Public Results - Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream266 Results Range From 9 To 841 items/sec938679612515418321224127029932835738641544447350253156058961864767670573476379282185020406080100

Based on OpenBenchmarking.org data, the selected test / test configuration (Neural Magic DeepSparse 1.1 - Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream) has an average run-time of 3 minutes. By default this test profile is set to run at least 3 times but may increase if the standard deviation exceeds pre-defined defaults or other calculations deem additional runs necessary for greater statistical accuracy of the result.

OpenBenchmarking.orgMinutesTime Required To Complete BenchmarkModel: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-StreamRun-Time246810Min: 2 / Avg: 2.79 / Max: 5

Does It Scale Well With Increasing Cores?

Yes, based on the automated analysis of the collected public benchmark data, this test / test settings does generally scale well with increasing CPU core counts. Data based on publicly available results for this test / test settings, separated by vendor, result divided by the reference CPU clock speed, grouped by matching physical CPU core count, and normalized against the smallest core count tested from each vendor for each CPU having a sufficient number of test samples and statistically significant data.

IntelAMDOpenBenchmarking.orgRelative Core Scaling To BaseNeural Magic DeepSparse CPU Core ScalingModel: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream48326412848121620

Tested CPU Architectures

This benchmark has been successfully tested on the below mentioned architectures. The CPU architectures listed is where successful OpenBenchmarking.org result uploads occurred, namely for helping to determine if a given test is compatible with various alternative CPU architectures.

CPU Architecture
Kernel Identifier
Verified On
Intel / AMD x86 64-bit
x86_64
(Many Processors)

Recent Test Results

OpenBenchmarking.org Results Compare

3 Systems - 270 Benchmark Results

AMD EPYC 7F32 8-Core - ASRockRack EPYCD8 - AMD Starship

Debian 11 - 5.10.0-10-amd64 - GNOME Shell 3.38.6

2 Systems - 270 Benchmark Results

AMD EPYC 7F32 8-Core - ASRockRack EPYCD8 - AMD Starship

Debian 11 - 5.10.0-10-amd64 - GNOME Shell 3.38.6

2 Systems - 334 Benchmark Results

2 x AMD EPYC 7601 32-Core - Dell 02MJ3T - AMD 17h

Ubuntu 22.04 - 5.15.0-40-generic - GNOME Shell 42.2

2 Systems - 334 Benchmark Results

2 x AMD EPYC 7601 32-Core - Dell 02MJ3T - AMD 17h

Ubuntu 22.04 - 5.15.0-40-generic - GNOME Shell 42.2

3 Systems - 184 Benchmark Results

AMD Ryzen 9 3900XT 12-Core - MSI MEG X570 GODLIKE - AMD Starship

Ubuntu 22.04 - 5.15.0-47-generic - GNOME Shell 42.2

1 System - 62 Benchmark Results

2 x AMD EPYC 7713 64-Core - AMD DAYTONA_X - AMD Starship

Ubuntu 22.04 - 5.15.0-47-generic - GNOME Shell 42.4

24 Systems - 193 Benchmark Results

AMD EPYC 9554 64-Core - AMD Titanite_4G - AMD Device 14a4

Ubuntu 22.10 - 6.0.0-060000rc3daily20220904-generic - GNOME Shell

2 Systems - 303 Benchmark Results

AMD Ryzen 9 7900X 12-Core - ASRock X670E PG Lightning - AMD Device 14d8

Ubuntu 22.10 - 5.19.0-23-generic - GNOME Shell 43.0

1 System - 28 Benchmark Results

AMD Ryzen 9 7950X 16-Core - ASRock X670E PG Lightning - AMD Device 14d8

Ubuntu 22.10 - 5.19.0-23-generic - GNOME Shell 43.0

20 Systems - 199 Benchmark Results

2 x AMD EPYC 7763 64-Core - AMD DAYTONA_X - AMD Starship

Ubuntu 22.10 - 6.0.0-060000rc3daily20220904-generic - GNOME Shell

2 Systems - 384 Benchmark Results

Intel Core i9-13900K - ASUS PRIME Z790-P WIFI - Intel Device 7a27

Ubuntu 22.10 - 5.19.0-23-generic - GNOME Shell 43.0

2 Systems - 168 Benchmark Results

AMD Ryzen 9 7950X 16-Core - ASUS ROG CROSSHAIR X670E HERO - AMD Device 14d8

Ubuntu 22.10 - 6.0.0-060000-generic - GNOME Shell 43.0

1 System - 384 Benchmark Results

Intel Core i9-13900K - ASUS PRIME Z790-P WIFI - Intel Device 7a27

Ubuntu 22.10 - 5.19.0-23-generic - GNOME Shell 43.0

1 System - 380 Benchmark Results

Intel Core i9-13900K - ASUS PRIME Z790-P WIFI - Intel Device 7a27

Ubuntu 22.10 - 5.19.0-23-generic - GNOME Shell 43.0

Most Popular Test Results

OpenBenchmarking.org Results Compare

4 Systems - 145 Benchmark Results

AMD Ryzen 7 5800X3D 8-Core - ASUS ROG CROSSHAIR VIII HERO - AMD Starship

Ubuntu 22.04 - 5.15.47+prerelease3723 - GNOME Shell 42.2

3 Systems - 46 Benchmark Results

AMD Ryzen Threadripper 3970X 32-Core - ASUS ROG ZENITH II EXTREME - AMD Starship

Ubuntu 22.04 - 5.19.0-051900rc7-generic - GNOME Shell 42.2

20 Systems - 199 Benchmark Results

Intel Xeon Platinum 8380 - Intel M50CYP2SB2U - Intel Ice Lake IEH

Ubuntu 22.10 - 6.0.0-060000rc3daily20220904-generic - GNOME Shell

3 Systems - 33 Benchmark Results

Intel Core i7-1065G7 - Dell 06CDVY - Intel Ice Lake-LP DRAM

Ubuntu 22.04 - 5.18.8-051808-generic - GNOME Shell 42.2

4 Systems - 73 Benchmark Results

AMD Ryzen Threadripper 3990X 64-Core - Gigabyte TRX40 AORUS PRO WIFI - AMD Starship

Ubuntu 22.10 - 6.0.0-060000rc7daily20220927-generic - GNOME Shell 43.0

4 Systems - 43 Benchmark Results

Intel Core i9-12900K - ASUS ROG STRIX Z690-E GAMING WIFI - Intel Alder Lake-S PCH

Ubuntu 22.04 - 5.17.0-1019-oem - GNOME Shell 42.2

3 Systems - 167 Benchmark Results

Intel Core i5-13600K - ASUS PRIME Z790-P WIFI - Intel Device 7a27

Ubuntu 22.04 - 6.0.0-060000rc1daily20220820-generic - GNOME Shell 42.2

2 Systems - 207 Benchmark Results

AMD Ryzen 7 PRO 6850U - LENOVO 21CM0001US - AMD Device 14b5

Ubuntu 22.10 - 6.0.0-060000-generic - GNOME Shell 43.0

3 Systems - 65 Benchmark Results

AMD Ryzen 7 PRO 6850U - LENOVO 21CM0001US - AMD Device 14b5

Ubuntu 22.10 - 6.0.0-060000rc2daily20220824-generic - GNOME Shell 42.4

2 Systems - 159 Benchmark Results

Intel Xeon E5-2609 v4 - MSI X99A RAIDER - Intel Xeon E7 v4

Ubuntu 20.04 - 5.9.0-050900rc6daily20200926-generic - GNOME Shell 3.36.2

2 Systems - 549 Benchmark Results

2 Systems - 168 Benchmark Results

AMD Ryzen 9 7950X 16-Core - ASUS ROG CROSSHAIR X670E HERO - AMD Device 14d8

Ubuntu 22.10 - 6.0.0-060000-generic - GNOME Shell 43.0

2 Systems - 384 Benchmark Results

Intel Core i9-13900K - ASUS PRIME Z790-P WIFI - Intel Device 7a27

Ubuntu 22.10 - 5.19.0-23-generic - GNOME Shell 43.0

2 Systems - 144 Benchmark Results

AMD Ryzen 7 7700X 8-Core - ASRock X670E PG Lightning - AMD Device 14d8

Ubuntu 22.04 - 5.17.0-1013-oem - GNOME Shell 42.2

4 Systems - 142 Benchmark Results

Intel Core i7-10700T - Logic Supply RXM-181 - Intel Comet Lake PCH

Ubuntu 22.04 - 5.15.0-48-generic - GNOME Shell 42.2

Find More Test Results