Neural Magic DeepSparse Benchmark - OpenBenchmarking.org

This is a benchmark of Neural Magic's DeepSparse using its built-in deepsparse.benchmark utility and various models from their SparseZoo (https://sparsezoo.neuralmagic.com/).

To run this test with the Phoronix Test Suite, the basic command is: phoronix-test-suite benchmark deepsparse.

* Uploading of benchmark result data to OpenBenchmarking.org is always optional (opt-in) via the Phoronix Test Suite for users wishing to share their results publicly.
** Data based on those opting to upload their test results to OpenBenchmarking.org and users enabling the opt-in anonymous statistics reporting while running benchmarks from an Internet-connected platform.
Data current as of 27 January 2023.

Revision History

pts/deepsparse-1.3.2 [View Source] Sun, 22 Jan 2023 19:05:03 GMT
Update against DeepSparse 1.3.2 upstream.

pts/deepsparse-1.0.1 [View Source] Thu, 13 Oct 2022 13:47:39 GMT
Initial commit of DeepSparse benchmark.

Performance Metrics

Neural Magic DeepSparse 1.3.2

Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream

OpenBenchmarking.org metrics for this test profile configuration based on 48 public results since 22 January 2023 with the latest data as of 1 February 2023.

Below is an overview of the generalized performance for components where there is sufficient statistically significant data based upon user-uploaded results. It is important to keep in mind particularly in the Linux/open-source space there can be vastly different OS configurations, with this overview intended to offer just general guidance as to the performance expectations.

Component

Percentile Rank

# Compatible Public Results

ms/batch (Average)

AMD Ryzen 9 7950X 16-Core

88th

AMD Ryzen 9 7900 12-Core

76th

46 ^{+/- 1}

Mid-Tier

75th

> 47

Intel Core i7-1165G7

67th

67 ^{+/- 5}

Intel Core i7-1185G7

63rd

69 ^{+/- 3}

AMD Ryzen 9 5900HX

51st

Median

50th

2 x AMD EPYC 9654 96-Core

44th

Intel Core i9-13900K

42nd

81 ^{+/- 1}

2 x Intel Xeon Platinum 8490H

32nd

83 ^{+/- 1}

Low-Tier

25th

> 88

2 x Intel Xeon Platinum 8380

17th

AMD Ryzen 7 PRO 6850U

17th

Intel Core i7-1065G7

9th

107

AMD Ryzen Threadripper 3990X 64-Core

3rd

113 ^{+/- 1}

Based on OpenBenchmarking.org data, the selected test / test configuration (Neural Magic DeepSparse 1.3.2 - Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream) has an average run-time of 2 minutes. By default this test profile is set to run at least 3 times but may increase if the standard deviation exceeds pre-defined defaults or other calculations deem additional runs necessary for greater statistical accuracy of the result.

Tested CPU Architectures

This benchmark has been successfully tested on the below mentioned architectures. The CPU architectures listed is where successful OpenBenchmarking.org result uploads occurred, namely for helping to determine if a given test is compatible with various alternative CPU architectures.

Ryzen 9 7900 3 Systems - 195 Benchmark Results	AMD Ryzen 9 7900 12-Core - Gigabyte B650M DS3H - AMD Device 14d8 Ubuntu 22.10 - 6.2.0-060200rc5daily20230129-generic - GNOME Shell 43.0
rocksdb deepsparse xeon sapphire rapids 3 Systems - 43 Benchmark Results	2 x Intel Xeon Platinum 8490H - Quanta Cloud S6Q-MB-MPS - Intel Device 1bce CentOS Stream 9 - 5.14.0-239.el9.x86_64 - GNOME Shell 40.10
rocksdb deepsparse epyc 3 Systems - 43 Benchmark Results	2 x AMD EPYC 9654 96-Core - AMD Titanite_4G - AMD Device 14a4 CentOS Stream 9 - 5.14.0-239.el9.x86_64 - GNOME Shell 40.10
8380 2023 Featured Vulkan Comparison	1.3.211 - 1.3.224
deepsparse 7950x Featured Graphics Comparison	AMD Radeon RX 6800
deepsparse tr 3 Systems - 36 Benchmark Results	AMD Ryzen Threadripper 3990X 64-Core - Gigabyte TRX40 AORUS PRO WIFI - AMD Starship Ubuntu 22.10 - 6.1.0-rc8-phx-mglru - GNOME Shell 43.0
icelake deep 3 Systems - 60 Benchmark Results	Intel Core i7-1065G7 - Dell 06CDVY - Intel Ice Lake-LP DRAM Ubuntu 22.04 - 5.18.8-051808-generic - GNOME Shell 42.2
ds key 3 Systems - 60 Benchmark Results	Intel Core i7-1185G7 - Dell 0DXP1F - Intel Tiger Lake-LP Ubuntu 22.04 - 5.18.8-051808-generic - GNOME Shell 42.2
ds lkey 3 Systems - 60 Benchmark Results	AMD Ryzen 7 PRO 6850U - LENOVO 21CM0001US - AMD Device 14b5 Ubuntu 22.10 - 6.1.0-060100rc2daily20221028-generic - GNOME Shell 43.0
new tests 4 Systems - 60 Benchmark Results	Intel Core i9-13900K - ASUS PRIME Z790-P WIFI - Intel Device 7a27 Ubuntu 22.04 - 6.0.0-060000-generic - GNOME Shell 42.4
deepsparse new 3 Systems - 36 Benchmark Results	AMD Ryzen 9 5900HX - ASUS G513QY v1.0 - AMD Renoir Ubuntu 22.10 - 5.19.0-28-generic - GNOME Shell 43.0
Intel Core i3-3220 Benchmarks 1 System - 8 Benchmark Results	Intel Core i3-3220 - Dell 042P49 - Intel Xeon E3-1200 v2 Linuxmint 20.1 - 5.4.0-136-generic - Cinnamon 4.8.6
Intel Core i3-3220 On Linuxmint 20.1 1 System - 5 Benchmark Results	Intel Core i3-3220 - Dell 042P49 - Intel Xeon E3-1200 v2 Linuxmint 20.1 - 5.4.0-136-generic - Cinnamon 4.8.6
Intel Core i3-3220 + llvmpipe + Dell 042P49 (A10 BIOS) 1 System - 4 Benchmark Results	Intel Core i3-3220 - Dell 042P49 - Intel Xeon E3-1200 v2 Linuxmint 20.1 - 5.4.0-136-generic - Cinnamon 4.8.6
deb11 3 Systems - 211 Benchmark Results	Intel Core i7-5960X - Gigabyte X99-UD4-CF - Intel Xeon E7 v3 Debian 11 - 5.10.0-10-amd64 - 1.0.2

deepsparse 7950x Featured Graphics Comparison	AMD Ryzen 9 7950X 16-Core - ASUS ROG CROSSHAIR X670E HERO - AMD Device 14d8 Ubuntu 22.10 - 5.19.0-29-generic - GNOME Shell 43.1
deepsparse new 3 Systems - 36 Benchmark Results	AMD Ryzen 9 5900HX - ASUS G513QY v1.0 - AMD Renoir Ubuntu 22.10 - 5.19.0-28-generic - GNOME Shell 43.0
new tests 4 Systems - 60 Benchmark Results	Intel Core i9-13900K - ASUS PRIME Z790-P WIFI - Intel Device 7a27 Ubuntu 22.04 - 6.0.0-060000-generic - GNOME Shell 42.4
icelake deep 3 Systems - 60 Benchmark Results	Intel Core i7-1065G7 - Dell 06CDVY - Intel Ice Lake-LP DRAM Ubuntu 22.04 - 5.18.8-051808-generic - GNOME Shell 42.2
rocksdb deepsparse xeon sapphire rapids 3 Systems - 43 Benchmark Results	2 x Intel Xeon Platinum 8490H - Quanta Cloud S6Q-MB-MPS - Intel Device 1bce CentOS Stream 9 - 5.14.0-239.el9.x86_64 - GNOME Shell 40.10
deepsparse tr 3 Systems - 36 Benchmark Results	AMD Ryzen Threadripper 3990X 64-Core - Gigabyte TRX40 AORUS PRO WIFI - AMD Starship Ubuntu 22.10 - 6.1.0-rc8-phx-mglru - GNOME Shell 43.0
Arch Linux Kernels 5 Systems - 96 Benchmark Results	AMD Ryzen 9 7950X 16-Core - ASUS ROG CROSSHAIR X670E HERO - AMD Device 14d8 EndeavourOS rolling - 5.15.89-1-lts - KDE Plasma 5.26.5
ds key 3 Systems - 60 Benchmark Results	Intel Core i7-1185G7 - Dell 0DXP1F - Intel Tiger Lake-LP Ubuntu 22.04 - 5.18.8-051808-generic - GNOME Shell 42.2
rocksdb deepsparse epyc 3 Systems - 43 Benchmark Results	2 x AMD EPYC 9654 96-Core - AMD Titanite_4G - AMD Device 14a4 CentOS Stream 9 - 5.14.0-239.el9.x86_64 - GNOME Shell 40.10
ds lkey 3 Systems - 60 Benchmark Results	AMD Ryzen 7 PRO 6850U - LENOVO 21CM0001US - AMD Device 14b5 Ubuntu 22.10 - 6.1.0-060100rc2daily20221028-generic - GNOME Shell 43.0
Ryzen 9 7900 3 Systems - 195 Benchmark Results	AMD Ryzen 9 7900 12-Core - Gigabyte B650M DS3H - AMD Device 14d8 Ubuntu 22.10 - 6.2.0-060200rc5daily20230129-generic - GNOME Shell 43.0
8380 2023 Featured Vulkan Comparison	2 x Intel Xeon Platinum 8380 - Intel M50CYP2SB2U - Intel Ice Lake IEH Ubuntu 22.10 - 6.2.0-rc5-phx-dodt - GNOME Shell
Xeon Platinum 8380 DODT Mitigation Impact 2 Systems - 86 Benchmark Results	2 x Intel Xeon Platinum 8380 - Intel M50CYP2SB2U - Intel Ice Lake IEH Ubuntu 22.10 - 6.2.0-rc5-phx-dodt - GNOME Shell
AM5 Noctua Cooling 2 Systems - 411 Benchmark Results	AMD Ryzen 9 7900 12-Core - Gigabyte B650M DS3H - AMD Device 14d8 Ubuntu 22.10 - 6.2.0-060200rc5daily20230129-generic - GNOME Shell 43.0

Neural Magic DeepSparse

Project Site

Source Repository

Test Created

Last Updated

Test Maintainer

Test Type

Average Install Time

Average Run Time

Test Dependencies

Accolades

Supported Platforms

Revision History

Suites Using This Test

Performance Metrics

Neural Magic DeepSparse 1.3.2

Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream

Tested CPU Architectures

Recent Test Results

Most Popular Test Results