Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations.

To run this test with the Phoronix Test Suite, the basic command is: phoronix-test-suite benchmark spark.

Project Site

spark.apache.org

Source Repository

github.com

Test Created

4 August 2022

Last Updated

20 January 2023

Test Maintainer

Michael Larabel 

Test Type

System

Average Install Time

4 Seconds

Average Run Time

18 Minutes, 7 Seconds

Test Dependencies

Java + Python

Accolades

20k+ Downloads

Supported Platforms

Public Result Uploads *Reported Installs **Reported Test Completions **Test Profile Page ViewsOpenBenchmarking.orgEventsApache Spark Popularity Statisticspts/spark2022.082022.092022.102022.112022.122023.012023.022023.032023.042023.052023.062023.072023.082023.092023.102023.113K6K9K12K15K
* Uploading of benchmark result data to OpenBenchmarking.org is always optional (opt-in) via the Phoronix Test Suite for users wishing to share their results publicly.
** Data based on those opting to upload their test results to OpenBenchmarking.org and users enabling the opt-in anonymous statistics reporting while running benchmarks from an Internet-connected platform.
Data updated weekly as of 28 November 2023.
2000000021.4%1000000022.4%100000033.6%4000000022.6%Row Count Option PopularityOpenBenchmarking.org
10031.1%200024.9%50022.5%100021.5%Partitions Option PopularityOpenBenchmarking.org

Revision History

pts/spark-1.0.1   [View Source]   Fri, 20 Jan 2023 07:57:46 GMT
Update download mirror for Spark 3.3.0. Closes: https://github.com/phoronix-test-suite/test-profiles/issues/265

pts/spark-1.0.0   [View Source]   Thu, 04 Aug 2022 04:35:20 GMT
Initial commit of Apache Spark benchmark.

Suites Using This Test

Database Test Suite

Server


Performance Metrics

Analyze Test Configuration:

Apache Spark 3.3

Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark

OpenBenchmarking.org metrics for this test profile configuration based on 412 public results since 4 August 2022 with the latest data as of 24 November 2023.

Below is an overview of the generalized performance for components where there is sufficient statistically significant data based upon user-uploaded results. It is important to keep in mind particularly in the Linux/open-source space there can be vastly different OS configurations, with this overview intended to offer just general guidance as to the performance expectations.

Component
Percentile Rank
# Compatible Public Results
Seconds (Average)
2 x AMD EPYC 7773X 64-Core
99th
10
16
2 x AMD EPYC 7713 64-Core
97th
4
18
ARMv8 Neoverse-N1 128-Core
96th
4
18
2 x AMD EPYC 7742 64-Core
95th
4
18
2 x AMD EPYC 75F3 32-Core
94th
4
24
AMD EPYC 7763 64-Core
92nd
4
32
2 x Intel Xeon Platinum 8380
91st
3
32
AMD Ryzen Threadripper 3990X 64-Core
89th
5
38 +/- 1
AMD Ryzen Threadripper PRO 5965WX 24-Cores
85th
5
50 +/- 1
Intel Core i9-13900K
83rd
5
52 +/- 1
AMD Ryzen Threadripper 3970X 32-Core
80th
3
56
AMD Ryzen 9 7950X 16-Core
80th
10
56 +/- 1
AMD Ryzen 9 7950X3D 16-Core
77th
8
59 +/- 1
AMD Ryzen Threadripper 2990WX 32-Core
76th
4
61
Mid-Tier
75th
> 62
2 x Intel Xeon Gold 5220R
74th
6
66
AMD Ryzen Threadripper 3960X 24-Core
71st
8
70 +/- 1
AMD Ryzen 9 7900X 12-Core
68th
11
73 +/- 2
AMD Ryzen 9 7900X3D 12-Core
65th
8
78 +/- 1
AMD EPYC 7551 32-Core
63rd
4
79 +/- 1
AMD Ryzen Threadripper 2970WX 24-Core
63rd
3
79 +/- 1
AMD Ryzen 9 5950X 16-Core
61st
23
81 +/- 6
AMD Ryzen 9 7900 12-Core
61st
4
82 +/- 1
Intel Core i9-12900K
61st
4
82 +/- 1
AMD Ryzen 9 3950X 16-Core
55th
3
88
Intel Core i9-10980XE
52nd
4
103
AMD Ryzen 7 7700X 8-Core
51st
8
107 +/- 1
Median
50th
107
AMD Ryzen Threadripper 2950X 16-Core
49th
3
109 +/- 1
AMD Ryzen 7 7700 8-Core
49th
5
112
AMD Ryzen 7 7800X3D 8-Core
46th
6
117 +/- 2
AMD Ryzen 9 3900X 12-Core
45th
3
122 +/- 1
Intel Core i5-12600K
45th
3
124
AMD Ryzen 7 5800X 8-Core
44th
3
136 +/- 1
AMD Ryzen 9 3900XT 12-Core
42nd
4
139
AMD Ryzen 7 5800X3D 8-Core
38th
6
147 +/- 2
AMD Ryzen 5 7600X 6-Core
38th
12
148 +/- 10
AMD Ryzen 9 5900HX
36th
7
157 +/- 5
Intel Xeon Silver 4216
34th
4
165
AMD EPYC 7F32 8-Core
30th
8
187 +/- 1
Low-Tier
25th
> 227
Intel Xeon E-2388G
24th
7
229
Intel Core i7-1280P
23rd
9
230 +/- 23
AMD Ryzen 7 PRO 5850U
22nd
7
231 +/- 3
AMD Ryzen 7 PRO 6850U
20th
8
242 +/- 6
Apple M1
18th
5
251 +/- 1
Intel Core i7-8700K
17th
3
265 +/- 1
AMD Ryzen 7 4800U
13th
3
305 +/- 3
AMD Ryzen 7 4700U
12th
7
317 +/- 15
Intel Xeon E-2336
11th
5
334 +/- 1
AMD Ryzen 5 4500U
9th
6
359 +/- 22
Intel Core i7-10700T
9th
3
374 +/- 1
Apple M2
5th
10
405 +/- 29
Intel Xeon E3-1275 v6
5th
3
405 +/- 1
Intel Xeon E3-1280 v5
4th
4
441 +/- 1
Intel Core i3-12100
2nd
4
584 +/- 6
Intel Core i7-1065G7
1st
3
709 +/- 50
OpenBenchmarking.orgDistribution Of Public Results - Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark412 Results Range From 9 To 1693 Seconds95091132173214255296337378419460501542583624665706747788829870911952993103410751116115711981239128013211362140314441485152615671608164916901731306090120150

Based on OpenBenchmarking.org data, the selected test / test configuration (Apache Spark 3.3 - Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark) has an average run-time of 14 minutes. By default this test profile is set to run at least 3 times but may increase if the standard deviation exceeds pre-defined defaults or other calculations deem additional runs necessary for greater statistical accuracy of the result.

OpenBenchmarking.orgMinutesTime Required To Complete BenchmarkRow Count: 1000000 - Partitions: 100 - Calculate Pi BenchmarkRun-Time1428425670Min: 2 / Avg: 13.87 / Max: 73

Based on public OpenBenchmarking.org results, the selected test / test configuration has an average standard deviation of 0.1%.

OpenBenchmarking.orgPercent, Fewer Is BetterAverage Deviation Between RunsRow Count: 1000000 - Partitions: 100 - Calculate Pi BenchmarkDeviation246810Min: 0 / Avg: 0.14 / Max: 2

Does It Scale Well With Increasing Cores?

Yes, based on the automated analysis of the collected public benchmark data, this test / test settings does generally scale well with increasing CPU core counts. Data based on publicly available results for this test / test settings, separated by vendor, result divided by the reference CPU clock speed, grouped by matching physical CPU core count, and normalized against the smallest core count tested from each vendor for each CPU having a sufficient number of test samples and statistically significant data.

IntelAMDOpenBenchmarking.orgRelative Core Scaling To BaseApache Spark CPU Core ScalingRow Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark4812162432641283691215

Tested CPU Architectures

This benchmark has been successfully tested on the below mentioned architectures. The CPU architectures listed is where successful OpenBenchmarking.org result uploads occurred, namely for helping to determine if a given test is compatible with various alternative CPU architectures.

CPU Architecture
Kernel Identifier
Verified On
Intel / AMD x86 64-bit
x86_64
(Many Processors)
ARMv8 64-bit
aarch64
ARMv8 Cortex-A72, ARMv8 Neoverse-N1, ARMv8 Neoverse-N1 128-Core, ARMv8 Neoverse-N1 64-Core, ARMv8 Neoverse-V1, ARMv8 Neoverse-V1 64-Core, Apple M1, Apple M2

Recent Test Results

OpenBenchmarking.org Results Compare

2 Systems - 452 Benchmark Results

ARMv8 Neoverse-N1 - GIGABYTE MP32-AR2-00 v01000100 - Ampere Computing LLC Altra PCI Root Complex A

Ubuntu 22.04 - 5.15.0-89-generic - 1.3.238

1 System - 484 Benchmark Results

ARMv8 Neoverse-N1 - GIGABYTE MP32-AR2-00 v01000100 - Ampere Computing LLC Device e100

Ubuntu 20.04 - 5.4.0-167-generic - 1.1.182

1 System - 112 Benchmark Results

AMD Ryzen 9 5950X 16-Core - ASUS TUF GAMING B550-PLUS WIFI II - AMD Starship

Fedora Linux 39 - 6.5.11-300.fc39.x86_64 - KDE Plasma 5.27.9

1 System - 7 Benchmark Results

ARMv8 Neoverse-V1 - Amazon EC2 r7gd.16xlarge - 496GB

Ubuntu 20.04 - 5.15.0-1036-aws - 1.1.182

2 Systems - 23 Benchmark Results

ARMv8 Neoverse-N1 - Amazon EC2 r6gd.16xlarge - 512GB

Ubuntu 20.04 - 5.15.0-1036-aws - 1.1.182

1 System - 260 Benchmark Results

Intel Xeon Platinum 8375C - Amazon EC2 m6i.4xlarge - Intel 440FX 82441FX PMC

Ubuntu 20.04 - 5.15.0-1049-aws - GNOME Shell 3.36.9

1 System - 251 Benchmark Results

Intel Xeon Platinum 8358 - QEMU Standard PC - Intel 440FX 82441FX PMC

Ubuntu 20.04 - 5.15.0-1042-oracle - 1.1.182

1 System - 256 Benchmark Results

AMD EPYC 7J13 64-Core - QEMU Standard PC - Intel 440FX 82441FX PMC

Ubuntu 20.04 - 5.15.0-1042-oracle - 1.1.182

1 System - 241 Benchmark Results

Intel Xeon Platinum 8375C - Amazon EC2 m6i.4xlarge - Intel 440FX 82441FX PMC

Ubuntu 20.04 - 5.15.0-1049-aws - GNOME Shell 3.36.9

1 System - 275 Benchmark Results

AMD EPYC 7R13 - Amazon EC2 m6a.4xlarge - Intel 440FX 82441FX PMC

Ubuntu 20.04 - 5.15.0-1048-aws - 1.1.182

1 System - 96 Benchmark Results

2 x Intel Xeon Platinum 8474C - Inspur NF5280M7 - Intel Device 1bce

Ubuntu 22.04 - 6.2.0-34-generic - X Server

1 System - 7 Benchmark Results

2 x Intel Xeon Platinum 8474C - Inspur NF5280M7 - Intel Device 1bce

Ubuntu 22.04 - 6.2.0-34-generic - X Server

1 System - 352 Benchmark Results

2 x Intel Xeon Platinum 8452Y - Lenovo SB27A92818 v06 - 8 x 32 GB DDR5-4800MT

AlmaLinux 9.2 - 5.14.0-284.30.1.el9_2.x86_64 - GCC 11.3.1 20221121

1 System - 115 Benchmark Results

2 x Intel Xeon Silver 4416+ - Dell PowerEdge R760 [0NH8MJ] - Intel Device 1bce

Ubuntu 18.04 - 5.4.0-150-generic - GNOME Shell 3.28.4

1 System - 105 Benchmark Results

2 x Intel Xeon Silver 4416+ - Dell PowerEdge R760 [0NH8MJ] - Intel Device 1bce

Ubuntu 18.04 - 5.4.0-150-generic - GNOME Shell 3.28.4

Most Popular Test Results

OpenBenchmarking.org Results Compare

4 Systems - 134 Benchmark Results

AMD Ryzen 9 5950X 16-Core - ASUS ROG CROSSHAIR VIII HERO - AMD Starship

Ubuntu 22.04 - 5.15.0-40-generic - GNOME Shell 42.2

2 Systems - 156 Benchmark Results

2 x AMD EPYC 75F3 32-Core - ASRockRack ROME2D16-2T - AMD Starship

Ubuntu 21.10 - 5.19.0-rc2-phx-mglru-v12 - GNOME Shell 40.5

3 Systems - 311 Benchmark Results

Intel Core i5-12600K - ASUS PRIME Z690-P WIFI D4 - Intel Device 7aa7

Ubuntu 22.04 - 5.19.0-051900rc6daily20220716-generic - GNOME Shell 42.1

2 Systems - 219 Benchmark Results

Apple M1 - Apple Mac mini - 8GB

Arch Linux ARM - 5.19.0-rc7-asahi-2-1-ARCH - KDE Plasma 5.25.4

2 Systems - 384 Benchmark Results

Intel Core i9-13900K - ASUS PRIME Z790-P WIFI - Intel Device 7a27

Ubuntu 22.10 - 5.19.0-23-generic - GNOME Shell 43.0

2 Systems - 112 Benchmark Results

AMD Ryzen 7 PRO 6850U - LENOVO 21CM0001US - AMD Device 14b5

Ubuntu 22.04 - 5.15.0-41-generic - GNOME Shell 42.2

2 Systems - 218 Benchmark Results

Intel Core i7-1280P - MSI MS-14C6 - Intel Alder Lake PCH

Arch Linux - 5.18.16-arch1-1 - KDE Plasma 5.25.4

3 Systems - 190 Benchmark Results

AMD Ryzen 7 PRO 6850U - LENOVO 21CM0001US - AMD Device 14b5

Arch rolling - 5.18.16-arch1-1 - KDE Plasma 5.25.4

2 Systems - 130 Benchmark Results

Intel Xeon Silver 4216 - TYAN S7100AG2NR - Intel Sky Lake-E DMI3 Registers

Debian 11 - 5.10.0-10-amd64 - X Server

2 Systems - 112 Benchmark Results

AMD Ryzen 7 PRO 5850U - HP 8A78 - AMD Renoir

Ubuntu 22.04 - 5.19.0-051900rc7-generic - GNOME Shell 42.2

2 Systems - 361 Benchmark Results

AMD Ryzen Threadripper 3960X 24-Core - MSI Creator TRX40 - AMD Starship

Ubuntu 22.04 - 5.19.0-051900rc7-generic - GNOME Shell 42.2
Find More Test Results