Apache Spark Benchmark - OpenBenchmarking.org

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations.

To run this test with the Phoronix Test Suite, the basic command is: phoronix-test-suite benchmark spark.

Performance Metrics

Apache Spark 3.3

Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe

OpenBenchmarking.org metrics for this test profile configuration based on 114 public results since 4 August 2022 with the latest data as of 10 August 2022.

Below is an overview of the generalized performance for components where there is sufficient statistically significant data based upon user-uploaded results. It is important to keep in mind particularly in the Linux/open-source space there can be vastly different OS configurations, with this overview intended to offer just general guidance as to the performance expectations.

Component

Percentile Rank

# Compatible Public Results

Seconds (Average)

2 x AMD EPYC 75F3 32-Core

99th

2.6 ^{+/- 0.1}

AMD Ryzen Threadripper 3990X 64-Core

94th

2.7 ^{+/- 0.2}

2 x Intel Xeon Platinum 8380

93rd

2.8

2 x AMD EPYC 7713 64-Core

86th

3.4 ^{+/- 0.2}

AMD Ryzen Threadripper PRO 5965WX 24-Cores

85th

3.4

Mid-Tier

75th

> 4.4

AMD Ryzen Threadripper 2990WX 32-Core

75th

4.4 ^{+/- 0.1}

Intel Core i9-12900K

72nd

5.0 ^{+/- 0.2}

AMD Ryzen 9 3950X 16-Core

64th

5.6

AMD Ryzen Threadripper 2970WX 24-Core

62nd

5.7 ^{+/- 0.1}

AMD EPYC 7551 32-Core

58th

6.3 ^{+/- 0.2}

AMD Ryzen Threadripper 2950X 16-Core

57th

6.8

AMD Ryzen 7 5800X3D 8-Core

51st

8.6

Median

50th

8.8

AMD Ryzen 9 5900HX

49th

8.9 ^{+/- 0.1}

Intel Xeon Silver 4216

43rd

9.2

AMD EPYC 7F32 8-Core

39th

11.5 ^{+/- 0.1}

AMD Ryzen 7 PRO 5850U

35th

13.0 ^{+/- 0.1}

AMD Ryzen 7 PRO 6850U

33rd

13.4 ^{+/- 0.4}

Low-Tier

25th

> 14.4

Apple M1

23rd

14.5

Intel Core i7-1280P

20th

14.8 ^{+/- 0.9}

Apple M2

15th

21.2 ^{+/- 0.3}

Intel Xeon E3-1275 v6

12th

22.6

Intel Core i3-12100

8th

28.2 ^{+/- 0.2}

Intel Core i7-1065G7

6th

38.2 ^{+/- 0.2}

ARMv8 Cortex-A72 4-Core

3rd

81.0 ^{+/- 0.4}

Based on OpenBenchmarking.org data, the selected test / test configuration (Apache Spark 3.3 - Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe) has an average run-time of 13 minutes. By default this test profile is set to run at least 3 times but may increase if the standard deviation exceeds pre-defined defaults or other calculations deem additional runs necessary for greater statistical accuracy of the result.

Tested CPU Architectures

This benchmark has been successfully tested on the below mentioned architectures. The CPU architectures listed is where successful OpenBenchmarking.org result uploads occurred, namely for helping to determine if a given test is compatible with various alternative CPU architectures.

Apple M1 Mac Mini 2 Systems - 219 Benchmark Results	Apple M1 - Apple Mac mini - 8GB Arch Linux ARM - 5.19.0-rc7-asahi-2-1-ARCH - KDE Plasma 5.25.4
spark 12900K Intel 4 Systems - 112 Benchmark Results	Intel Core i9-12900K - ASUS ROG STRIX Z690-E GAMING WIFI - Intel Alder Lake-S PCH Ubuntu 22.04 - 5.19.0-051900-generic - GNOME Shell 42.2
august 4 Systems - 134 Benchmark Results	AMD Ryzen 9 5950X 16-Core - ASUS ROG CROSSHAIR VIII HERO - AMD Starship Ubuntu 22.04 - 5.15.0-40-generic - GNOME Shell 42.2
3990X spark 3 Systems - 112 Benchmark Results	AMD Ryzen Threadripper 3990X 64-Core - Gigabyte TRX40 AORUS PRO WIFI - AMD Starship Ubuntu 22.04 - 5.19.0-051900rc7-generic - GNOME Shell 42.2
apache spark icelake 3 Systems - 112 Benchmark Results	Intel Core i7-1065G7 - Dell 06CDVY - Intel Ice Lake-LP DRAM Ubuntu 22.04 - 5.18.8-051808-generic - GNOME Shell 42.2
75f3 2p august 2 Systems - 156 Benchmark Results	2 x AMD EPYC 75F3 32-Core - ASRockRack ROME2D16-2T - AMD Starship Ubuntu 21.10 - 5.19.0-rc2-phx-mglru-v12 - GNOME Shell 40.5
Apple M1 Mac Mini 1 System - 219 Benchmark Results	Apple M1 - Apple Mac mini - 8GB Arch Linux ARM - 5.19.0-rc7-asahi-2-1-ARCH - KDE Plasma 5.25.4
spark tr 2 Systems - 120 Benchmark Results	AMD Ryzen Threadripper 2990WX 32-Core - ASUS ROG ZENITH EXTREME - AMD 17h Ubuntu 22.04 - 5.19.0-051900rc7-generic - GNOME Shell 42.2
spark rembrandty 2 Systems - 112 Benchmark Results	AMD Ryzen 7 PRO 6850U - LENOVO 21CM0001US - AMD Device 14b5 Ubuntu 22.04 - 5.15.0-41-generic - GNOME Shell 42.2
5850u spark 2 Systems - 112 Benchmark Results	AMD Ryzen 7 PRO 5850U - HP 8A78 - AMD Renoir Ubuntu 22.04 - 5.19.0-051900rc7-generic - GNOME Shell 42.2
alderlake-p-spark 3 Systems - 112 Benchmark Results	Intel Core i7-1280P - MSI MS-14C6 - Intel Alder Lake PCH Ubuntu 22.04 - 5.19.0-051900rc7daily20220723-generic - GNOME Shell 42.2
xeon aug 2 Systems - 130 Benchmark Results	Intel Xeon Silver 4216 - TYAN S7100AG2NR - Intel Sky Lake-E DMI3 Registers Debian 11 - 5.10.0-10-amd64 - X Server
new ep 2 Systems - 130 Benchmark Results	AMD EPYC 7F32 8-Core - ASRockRack EPYCD8 - AMD Starship Debian 11 - 5.10.0-10-amd64 - GNOME Shell 3.38.6
5950xGentoo-ApacheSpark-all 1 System - 112 Benchmark Results	AMD Ryzen 9 5950X 16-Core - Gigabyte X570 AORUS ELITE WIFI - AMD Starship Gentoo 2.8 - 5.19.0-gentoo-harambe-edition - KDE Plasma 5.25.4
apache spark 5800X3D 3 Systems - 112 Benchmark Results	AMD Ryzen 7 5800X3D 8-Core - ASUS ROG CROSSHAIR VIII HERO - AMD Starship Ubuntu 22.04 - 5.19.0-rc6-drm-next-28july - GNOME Shell 42.2

spark svt 4 Systems - 120 Benchmark Results	AMD EPYC 7551 32-Core - GIGABYTE MZ31-AR0-00 v01010101 - AMD 17h Debian 11 - 5.10.0-9-amd64 - GCC 10.2.1 20210110
lappy M1 3 Systems - 211 Benchmark Results	Apple M1 - Apple Mac mini - 8GB Arch Linux ARM - 5.19.0-rc7-asahi-2-1-ARCH - KDE Plasma 5.25.4
Apache Spark AMD EPYC 3 Systems - 76 Benchmark Results	2 x AMD EPYC 7713 64-Core - AMD DAYTONA_X - AMD Starship Ubuntu 22.04 - 5.19.0-051900daily20220803-generic - GNOME Shell 42.2
apache spark 5800X3D 3 Systems - 112 Benchmark Results	AMD Ryzen 7 5800X3D 8-Core - ASUS ROG CROSSHAIR VIII HERO - AMD Starship Ubuntu 22.04 - 5.19.0-rc6-drm-next-28july - GNOME Shell 42.2
5850u spark 2 Systems - 112 Benchmark Results	AMD Ryzen 7 PRO 5850U - HP 8A78 - AMD Renoir Ubuntu 22.04 - 5.19.0-051900rc7-generic - GNOME Shell 42.2
alderlake-p-spark 3 Systems - 112 Benchmark Results	Intel Core i7-1280P - MSI MS-14C6 - Intel Alder Lake PCH Ubuntu 22.04 - 5.19.0-051900rc7daily20220723-generic - GNOME Shell 42.2
spark tr 2 Systems - 120 Benchmark Results	AMD Ryzen Threadripper 2990WX 32-Core - ASUS ROG ZENITH EXTREME - AMD 17h Ubuntu 22.04 - 5.19.0-051900rc7-generic - GNOME Shell 42.2
xeon aug 2 Systems - 130 Benchmark Results	Intel Xeon Silver 4216 - TYAN S7100AG2NR - Intel Sky Lake-E DMI3 Registers Debian 11 - 5.10.0-10-amd64 - X Server
apache spark 5900HX Featured Graphics Comparison	AMD Ryzen 9 5900HX - ASUS G513QY v1.0 - AMD Renoir Ubuntu 22.10 - 5.15.0-27-generic - GNOME Shell 42.3.1
spark rembrandty 2 Systems - 112 Benchmark Results	AMD Ryzen 7 PRO 6850U - LENOVO 21CM0001US - AMD Device 14b5 Ubuntu 22.04 - 5.15.0-41-generic - GNOME Shell 42.2
new ep 2 Systems - 130 Benchmark Results	AMD EPYC 7F32 8-Core - ASRockRack EPYCD8 - AMD Starship Debian 11 - 5.10.0-10-amd64 - GNOME Shell 3.38.6
75f3 2p august 2 Systems - 156 Benchmark Results	2 x AMD EPYC 75F3 32-Core - ASRockRack ROME2D16-2T - AMD Starship Ubuntu 21.10 - 5.19.0-rc2-phx-mglru-v12 - GNOME Shell 40.5
apache spark icelake 3 Systems - 112 Benchmark Results	Intel Core i7-1065G7 - Dell 06CDVY - Intel Ice Lake-LP DRAM Ubuntu 22.04 - 5.18.8-051808-generic - GNOME Shell 42.2
3990X spark 3 Systems - 112 Benchmark Results	AMD Ryzen Threadripper 3990X 64-Core - Gigabyte TRX40 AORUS PRO WIFI - AMD Starship Ubuntu 22.04 - 5.19.0-051900rc7-generic - GNOME Shell 42.2
august 4 Systems - 134 Benchmark Results	AMD Ryzen 9 5950X 16-Core - ASUS ROG CROSSHAIR VIII HERO - AMD Starship Ubuntu 22.04 - 5.15.0-40-generic - GNOME Shell 42.2

Apache Spark

Project Site

Source Repository

Test Created

Test Maintainer

Test Type

Average Install Time

Average Run Time

Test Dependencies

Accolades

Supported Platforms

Revision History

Performance Metrics

Apache Spark 3.3

Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe

Tested CPU Architectures

Recent Test Results

Most Popular Test Results