Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations.

To run this test with the Phoronix Test Suite, the basic command is: phoronix-test-suite benchmark spark.

Project Site

spark.apache.org

Source Repository

github.com

Test Created

4 August 2022

Test Maintainer

Michael Larabel 

Test Type

System

Average Install Time

5 Seconds

Average Run Time

9 Minutes, 24 Seconds

Test Dependencies

Java + Python

Accolades

Recently Created Test Profile

Supported Platforms


2000000023.3%1000000024.2%100000028.9%4000000023.6%Row Count Option PopularityOpenBenchmarking.org
10026.6%200025.7%50024.1%100023.6%Partitions Option PopularityOpenBenchmarking.org

Revision History

pts/spark-1.0.0   [View Source]   Thu, 04 Aug 2022 04:35:20 GMT
Initial commit of Apache Spark benchmark.


Performance Metrics

Analyze Test Configuration:

Apache Spark 3.3

Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe

OpenBenchmarking.org metrics for this test profile configuration based on 114 public results since 4 August 2022 with the latest data as of 10 August 2022.

Below is an overview of the generalized performance for components where there is sufficient statistically significant data based upon user-uploaded results. It is important to keep in mind particularly in the Linux/open-source space there can be vastly different OS configurations, with this overview intended to offer just general guidance as to the performance expectations.

Component
Percentile Rank
# Compatible Public Results
Seconds (Average)
Mid-Tier
75th
> 4.4
72nd
4
5.0 +/- 0.2
58th
4
6.3 +/- 0.2
Median
50th
8.8
49th
7
8.9 +/- 0.1
39th
4
11.5 +/- 0.1
35th
4
13.0 +/- 0.1
33rd
8
13.4 +/- 0.4
Low-Tier
25th
> 14.4
23rd
5
14.5
20th
6
14.8 +/- 0.9
15th
4
21.2 +/- 0.3
8th
4
28.2 +/- 0.2
6th
3
38.2 +/- 0.2
3rd
4
81.0 +/- 0.4
OpenBenchmarking.orgDistribution Of Public Results - Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe58 Results Range From 2 To 39 Seconds261014182226303438424650510152025

Based on OpenBenchmarking.org data, the selected test / test configuration (Apache Spark 3.3 - Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe) has an average run-time of 13 minutes. By default this test profile is set to run at least 3 times but may increase if the standard deviation exceeds pre-defined defaults or other calculations deem additional runs necessary for greater statistical accuracy of the result.

OpenBenchmarking.orgMinutesTime Required To Complete BenchmarkRow Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark Using DataframeRun-Time612182430Min: 3 / Avg: 13.12 / Max: 24

Tested CPU Architectures

This benchmark has been successfully tested on the below mentioned architectures. The CPU architectures listed is where successful OpenBenchmarking.org result uploads occurred, namely for helping to determine if a given test is compatible with various alternative CPU architectures.

CPU Architecture
Kernel Identifier
Verified On
Intel / AMD x86 64-bit
x86_64
(Many Processors)
ARMv8 64-bit
aarch64
ARMv8 Cortex-A72 4-Core, ARMv8 Neoverse-N1, Apple M1, Apple M2

Recent Test Results

OpenBenchmarking.org Results Compare

2 Systems - 219 Benchmark Results

Apple M1 - Apple Mac mini - 8GB

Arch Linux ARM - 5.19.0-rc7-asahi-2-1-ARCH - KDE Plasma 5.25.4

4 Systems - 112 Benchmark Results

Intel Core i9-12900K - ASUS ROG STRIX Z690-E GAMING WIFI - Intel Alder Lake-S PCH

Ubuntu 22.04 - 5.19.0-051900-generic - GNOME Shell 42.2

4 Systems - 134 Benchmark Results

AMD Ryzen 9 5950X 16-Core - ASUS ROG CROSSHAIR VIII HERO - AMD Starship

Ubuntu 22.04 - 5.15.0-40-generic - GNOME Shell 42.2

3 Systems - 112 Benchmark Results

AMD Ryzen Threadripper 3990X 64-Core - Gigabyte TRX40 AORUS PRO WIFI - AMD Starship

Ubuntu 22.04 - 5.19.0-051900rc7-generic - GNOME Shell 42.2

3 Systems - 112 Benchmark Results

Intel Core i7-1065G7 - Dell 06CDVY - Intel Ice Lake-LP DRAM

Ubuntu 22.04 - 5.18.8-051808-generic - GNOME Shell 42.2

2 Systems - 156 Benchmark Results

2 x AMD EPYC 75F3 32-Core - ASRockRack ROME2D16-2T - AMD Starship

Ubuntu 21.10 - 5.19.0-rc2-phx-mglru-v12 - GNOME Shell 40.5

1 System - 219 Benchmark Results

Apple M1 - Apple Mac mini - 8GB

Arch Linux ARM - 5.19.0-rc7-asahi-2-1-ARCH - KDE Plasma 5.25.4

2 Systems - 120 Benchmark Results

AMD Ryzen Threadripper 2990WX 32-Core - ASUS ROG ZENITH EXTREME - AMD 17h

Ubuntu 22.04 - 5.19.0-051900rc7-generic - GNOME Shell 42.2

2 Systems - 112 Benchmark Results

AMD Ryzen 7 PRO 6850U - LENOVO 21CM0001US - AMD Device 14b5

Ubuntu 22.04 - 5.15.0-41-generic - GNOME Shell 42.2

2 Systems - 112 Benchmark Results

AMD Ryzen 7 PRO 5850U - HP 8A78 - AMD Renoir

Ubuntu 22.04 - 5.19.0-051900rc7-generic - GNOME Shell 42.2

3 Systems - 112 Benchmark Results

Intel Core i7-1280P - MSI MS-14C6 - Intel Alder Lake PCH

Ubuntu 22.04 - 5.19.0-051900rc7daily20220723-generic - GNOME Shell 42.2

2 Systems - 130 Benchmark Results

Intel Xeon Silver 4216 - TYAN S7100AG2NR - Intel Sky Lake-E DMI3 Registers

Debian 11 - 5.10.0-10-amd64 - X Server

2 Systems - 130 Benchmark Results

AMD EPYC 7F32 8-Core - ASRockRack EPYCD8 - AMD Starship

Debian 11 - 5.10.0-10-amd64 - GNOME Shell 3.38.6

1 System - 112 Benchmark Results

AMD Ryzen 9 5950X 16-Core - Gigabyte X570 AORUS ELITE WIFI - AMD Starship

Gentoo 2.8 - 5.19.0-gentoo-harambe-edition - KDE Plasma 5.25.4

3 Systems - 112 Benchmark Results

AMD Ryzen 7 5800X3D 8-Core - ASUS ROG CROSSHAIR VIII HERO - AMD Starship

Ubuntu 22.04 - 5.19.0-rc6-drm-next-28july - GNOME Shell 42.2

Most Popular Test Results

OpenBenchmarking.org Results Compare

4 Systems - 120 Benchmark Results

AMD EPYC 7551 32-Core - GIGABYTE MZ31-AR0-00 v01010101 - AMD 17h

Debian 11 - 5.10.0-9-amd64 - GCC 10.2.1 20210110

3 Systems - 211 Benchmark Results

Apple M1 - Apple Mac mini - 8GB

Arch Linux ARM - 5.19.0-rc7-asahi-2-1-ARCH - KDE Plasma 5.25.4

3 Systems - 76 Benchmark Results

2 x AMD EPYC 7713 64-Core - AMD DAYTONA_X - AMD Starship

Ubuntu 22.04 - 5.19.0-051900daily20220803-generic - GNOME Shell 42.2

3 Systems - 112 Benchmark Results

AMD Ryzen 7 5800X3D 8-Core - ASUS ROG CROSSHAIR VIII HERO - AMD Starship

Ubuntu 22.04 - 5.19.0-rc6-drm-next-28july - GNOME Shell 42.2

2 Systems - 112 Benchmark Results

AMD Ryzen 7 PRO 5850U - HP 8A78 - AMD Renoir

Ubuntu 22.04 - 5.19.0-051900rc7-generic - GNOME Shell 42.2

3 Systems - 112 Benchmark Results

Intel Core i7-1280P - MSI MS-14C6 - Intel Alder Lake PCH

Ubuntu 22.04 - 5.19.0-051900rc7daily20220723-generic - GNOME Shell 42.2

2 Systems - 120 Benchmark Results

AMD Ryzen Threadripper 2990WX 32-Core - ASUS ROG ZENITH EXTREME - AMD 17h

Ubuntu 22.04 - 5.19.0-051900rc7-generic - GNOME Shell 42.2

2 Systems - 130 Benchmark Results

Intel Xeon Silver 4216 - TYAN S7100AG2NR - Intel Sky Lake-E DMI3 Registers

Debian 11 - 5.10.0-10-amd64 - X Server

2 Systems - 112 Benchmark Results

AMD Ryzen 7 PRO 6850U - LENOVO 21CM0001US - AMD Device 14b5

Ubuntu 22.04 - 5.15.0-41-generic - GNOME Shell 42.2

2 Systems - 130 Benchmark Results

AMD EPYC 7F32 8-Core - ASRockRack EPYCD8 - AMD Starship

Debian 11 - 5.10.0-10-amd64 - GNOME Shell 3.38.6

2 Systems - 156 Benchmark Results

2 x AMD EPYC 75F3 32-Core - ASRockRack ROME2D16-2T - AMD Starship

Ubuntu 21.10 - 5.19.0-rc2-phx-mglru-v12 - GNOME Shell 40.5

3 Systems - 112 Benchmark Results

Intel Core i7-1065G7 - Dell 06CDVY - Intel Ice Lake-LP DRAM

Ubuntu 22.04 - 5.18.8-051808-generic - GNOME Shell 42.2

3 Systems - 112 Benchmark Results

AMD Ryzen Threadripper 3990X 64-Core - Gigabyte TRX40 AORUS PRO WIFI - AMD Starship

Ubuntu 22.04 - 5.19.0-051900rc7-generic - GNOME Shell 42.2

4 Systems - 134 Benchmark Results

AMD Ryzen 9 5950X 16-Core - ASUS ROG CROSSHAIR VIII HERO - AMD Starship

Ubuntu 22.04 - 5.15.0-40-generic - GNOME Shell 42.2

Find More Test Results