tr5 spark svt av1

Benchmarks for a future article.

Compare your own system(s) to this result file with the Phoronix Test Suite by running the command: phoronix-test-suite benchmark 2208075-NE-TR5SPARKS30
Jump To Table - Results

View

Do Not Show Noisy Results
Do Not Show Results With Incomplete Data
Do Not Show Results With Little Change/Spread
List Notable Results

Statistics

Show Overall Harmonic Mean(s)
Show Overall Geometric Mean
Show Wins / Losses Counts (Pie Chart)
Normalize Results
Remove Outliers Before Calculating Averages

Graph Settings

Force Line Graphs Where Applicable
Convert To Scalar Where Applicable
Prefer Vertical Bar Graphs

Multi-Way Comparison

Condense Multi-Option Tests Into Single Result Graphs

Table

Show Detailed System Result Table

Run Management

Highlight
Result
Hide
Result
Result
Identifier
View Logs
Performance Per
Dollar
Date
Run
  Test
  Duration
A
August 06 2022
  4 Hours, 9 Minutes
B
August 06 2022
  3 Hours, 59 Minutes
BB
August 07 2022
  5 Hours, 12 Minutes
Invert Hiding All Results Option
  4 Hours, 27 Minutes

Only show results where is faster than
Only show results matching title/arguments (delimit multiple options with a comma):
Do not show results matching title/arguments (delimit multiple options with a comma):


tr5 spark svt av1OpenBenchmarking.orgPhoronix Test SuiteAMD Ryzen Threadripper PRO 5965WX 24-Cores @ 3.80GHz (24 Cores / 48 Threads)ASUS Pro WS WRX80E-SAGE SE WIFI (1003 BIOS)AMD Starship/Matisse64GB1000GB Western Digital WDS100T1X0E-00AFY0ASUS NVIDIA NV106 2GBAMD Starship/MatisseVA24312 x Intel 10G X550T + Intel Wi-Fi 6 AX200Ubuntu 22.045.19.0-051900rc7-generic (x86_64)GNOME Shell 42.2X Server 1.21.1.3 + Waylandnouveau4.3 Mesa 22.0.11.2.204GCC 11.2.0ext41920x1080ProcessorMotherboardChipsetMemoryDiskGraphicsAudioMonitorNetworkOSKernelDesktopDisplay ServerDisplay DriverOpenGLVulkanCompilerFile-SystemScreen ResolutionTr5 Spark Svt Av1 BenchmarksSystem Logs- Transparent Huge Pages: madvise- --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-11-gBFGDP/gcc-11-11.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-gBFGDP/gcc-11-11.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v - Scaling Governor: acpi-cpufreq schedutil (Boost: Enabled) - CPU Microcode: 0xa008203- OpenJDK Runtime Environment (build 11.0.15+10-Ubuntu-0ubuntu0.22.04.1)- Python 3.10.4- itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling + srbds: Not affected + tsx_async_abort: Not affected

Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - Inner Join Test TimeABBB61218243013.1314.2424.06

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - Inner Join Test TimeBABB112233445526.1127.3247.29

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - Repartition Test TimeBABB91827364522.9523.6541.21

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Inner Join Test TimeBABB61218243013.5015.1224.17

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - Repartition Test TimeBABB102030405023.8424.3342.30

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - Broadcast Inner Join Test TimeBABB36912157.06000000011.24185690912.333389834

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - Broadcast Inner Join Test TimeBABB61218243013.4813.9623.51

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - Broadcast Inner Join Test TimeABBB102030405026.2728.4445.78

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - Repartition Test TimeBABB102030405024.3224.3642.34

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - Inner Join Test TimeBABB61218243014.6317.6425.42

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - Repartition Test TimeBABB102030405024.4526.7042.47

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - Repartition Test TimeBABB51015202512.3813.0621.38

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Broadcast Inner Join Test TimeBABB61218243013.7416.0123.68

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - Inner Join Test TimeBABB61218243014.1714.6924.23

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - Broadcast Inner Join Test TimeABBB51015202513.2313.8022.58

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - Repartition Test TimeABBB51015202512.3712.3921.03

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - Broadcast Inner Join Test TimeBABB102030405026.8127.6045.42

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - Broadcast Inner Join Test TimeBABB102030405025.3926.0342.82

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Repartition Test TimeBABB51015202512.6514.0521.32

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - Inner Join Test TimeBABB102030405026.4726.8044.62

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - Broadcast Inner Join Test TimeBABB61218243014.4717.4224.32

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - Repartition Test TimeBABB51015202513.0815.7521.92

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - Inner Join Test TimeBABB36912157.76000000011.79095691212.930379514

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - Inner Join Test TimeBABB102030405026.7927.7644.56

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - SHA-512 Benchmark TimeABBB142842567036.7037.2860.83

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - SHA-512 Benchmark TimeBABB142842567038.4940.7463.38

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Broadcast Inner Join Test TimeBABB36912157.48000000011.72111193112.264914309

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - Inner Join Test TimeBABB102030405027.5628.2544.80

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - Broadcast Inner Join Test TimeBABB102030405028.3328.7246.03

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - SHA-512 Benchmark TimeBABB142842567037.9938.8261.57

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - SHA-512 Benchmark TimeBABB81624324020.3724.1832.98

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Inner Join Test TimeBABB36912157.76000000011.42813491912.518076203

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - SHA-512 Benchmark TimeABBB81624324020.4620.5432.75

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - SHA-512 Benchmark TimeBABB132639526537.5837.7060.13

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Repartition Test TimeBABB36912157.1200000007.85000000011.383497001

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Repartition Test TimeBABB36912156.99000000010.12849683911.143807954

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Broadcast Inner Join Test TimeBABB36912157.950000008.3900000012.65896058

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - SHA-512 Benchmark TimeBABB81624324020.5324.4132.62

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - Repartition Test TimeBABB36912157.0300000009.79000000011.130497241

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Inner Join Test TimeBABB36912158.3100000008.59000000013.148685287

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - Broadcast Inner Join Test TimeBABB36912157.4100000007.85000000011.585953767

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - SHA-512 Benchmark TimeBABB81624324020.9821.2732.55

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - Repartition Test TimeBABB36912156.9200000007.56000000010.667978128

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - Inner Join Test TimeBABB36912158.0300000008.21000000012.207016521

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - Group By Test TimeBABB81624324022.2623.4633.39

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Broadcast Inner Join Test TimeBABB0.55011.10021.65032.20042.75051.6391141811.9690989852.444848775

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - SHA-512 Benchmark TimeBABB4812162011.5912.4017.11

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - SHA-512 Benchmark TimeBABB4812162011.7112.1817.27

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - SHA-512 Benchmark TimeBABB4812162012.0415.4117.61

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - Group By Test TimeBABB81624324022.8724.7733.34

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - Group By Test TimeBABB81624324022.8426.5833.11

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - Group By Test TimeBABB81624324022.7824.0332.59

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - SHA-512 Benchmark TimeBABB4812162011.9415.1317.07

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Broadcast Inner Join Test TimeBABB0.37510.75021.12531.50041.87551.2600000001.4711850981.667104916

SVT-AV1

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 8 - Input: Bosphorus 4KBABB153045607566.7359.2551.371. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 10 - Input: Bosphorus 4KBABB20406080100111.01102.1286.161. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 12 - Input: Bosphorus 4KBABB306090120150139.51129.14108.381. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - Group By Test TimeBABB36912158.9500000009.23000000011.503199565

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - Group By Test TimeBABB36912159.3500000009.91215491411.850374914

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Inner Join Test TimeBABB0.51891.03781.55672.07562.59451.8400000001.9599524232.306298059

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Inner Join Test TimeBABB0.57481.14961.72442.29922.8742.0400000002.1687180632.554630639

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Group By Test TimeBABB36912159.38000000010.85758825711.665361027

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Repartition Test TimeBABB0.57131.14261.71392.28522.85652.0500000002.1400000002.538924266

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - Group By Test TimeBABB36912159.51000000010.59503967511.759991319

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Repartition Test TimeBABB0.53031.06061.59092.12122.65151.9100000001.9700000002.356867444

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - Group By Test TimeBABB2468106.556.937.92

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Group By Test TimeBABB2468107.037.368.43

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Inner Join Test TimeBABB0.6931.3862.0792.7723.4652.5767158782.8500000003.080000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Group By Test TimeBABB2468106.8100000007.2400000008.059610009

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Repartition Test TimeBABB0.64341.28681.93022.57363.2172.4200000002.4571578092.859752415

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - Group By Test TimeBABB2468106.5700000007.1569716527.740000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Inner Join Test TimeBABB0.4640.9281.3921.8562.321.7600000001.8861240582.062209171

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Repartition Test TimeABBB0.49150.9831.47451.9662.45751.8900000001.8900000002.184657049

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Broadcast Inner Join Test TimeABBB0.60081.20161.80242.40323.0042.322.492.67

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Broadcast Inner Join Test TimeBABB0.31830.63660.95491.27321.59151.230000001.330000001.41487791

SVT-AV1

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 10 - Input: Bosphorus 1080pBABB70140210280350341.35309.01298.101. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 12 - Input: Bosphorus 1080pBABB100200300400500461.47418.67411.291. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - SHA-512 Benchmark TimeBABB0.97971.95942.93913.91884.89853.9000000004.1600000004.354368413

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - SHA-512 Benchmark TimeBABB0.92821.85642.78463.71284.6413.7053393093.9700000004.125141619

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Group By Test TimeBABB1.0212.0423.0634.0845.1054.1500000004.3100000004.537829999

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - SHA-512 Benchmark TimeBBBA0.85731.71462.57193.42924.28653.510000003.774602653.81000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Group By Test TimeBABB0.99191.98382.97573.96764.95954.0800000004.1792439594.408564623

SVT-AV1

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 8 - Input: Bosphorus 1080pBABB4080120160200161.39152.98149.401. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Group By Test TimeBABB0.96021.92042.88063.84084.8014.0100000004.0281825324.267736603

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - SHA-512 Benchmark TimeBBBA0.84911.69822.54733.39644.24553.5500000003.7525238303.773601794

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Group By Test TimeABBB1.12682.25363.38044.50725.6344.7182113244.7400000005.008039519

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - Calculate Pi Benchmark Using DataframeBBBA0.78281.56562.34843.13123.9143.3500000003.3900000003.479319347

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Calculate Pi BenchmarkBABB112233445549.0650.7550.79

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Calculate Pi Benchmark Using DataframeABBB0.77921.55842.33763.11683.8963.3474045573.4100000003.463213224

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - Calculate Pi Benchmark Using DataframeABBB0.7831.5662.3493.1323.9153.3700000003.4039372613.480000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Calculate Pi Benchmark Using DataframeBBBA0.7651.532.2953.063.8253.3100000003.3313364023.400000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - Calculate Pi BenchmarkBABB122436486050.0750.2751.41

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - Calculate Pi BenchmarkBABB112233445549.5349.9450.72

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - Calculate Pi Benchmark Using DataframeABBB0.7741.5482.3223.0963.873.363.443.44

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - Calculate Pi Benchmark Using DataframeBABB0.7741.5482.3223.0963.873.363.423.44

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - Calculate Pi Benchmark Using DataframeBBBA0.77631.55262.32893.10523.88153.373.383.45

SVT-AV1

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 4 - Input: Bosphorus 1080pBBBA2468107.4847.3877.3121. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - Calculate Pi BenchmarkBABB112233445549.7050.5250.85

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - Calculate Pi BenchmarkABBB122436486049.9050.0451.04

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - Calculate Pi Benchmark Using DataframeBABB0.7741.5482.3223.0963.873.373.393.44

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - Calculate Pi BenchmarkABBB112233445549.7850.0450.73

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Calculate Pi BenchmarkBBBA112233445549.2949.8150.22

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Calculate Pi Benchmark Using DataframeABBB0.76951.5392.30853.0783.84753.363.383.42

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - Calculate Pi BenchmarkBBBA112233445549.7950.0450.61

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Calculate Pi Benchmark Using DataframeABBB0.76731.53462.30193.06923.83653.3600000003.3903942883.410000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - Calculate Pi Benchmark Using DataframeABBB0.7741.5482.3223.0963.873.393.403.44

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Calculate Pi BenchmarkBBAB112233445549.7450.4350.47

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - Calculate Pi BenchmarkBBBA112233445549.7149.7550.39

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Calculate Pi BenchmarkBBBA112233445549.3249.4649.99

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Calculate Pi BenchmarkBBBA112233445550.0750.1550.71

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - Calculate Pi BenchmarkBABB112233445549.5249.7450.12

SVT-AV1

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 4 - Input: Bosphorus 4KABBB0.64551.2911.93652.5823.22752.8692.8562.8361. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - Calculate Pi BenchmarkBBBA112233445549.9450.2250.50

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Calculate Pi BenchmarkABBB112233445550.0750.3450.58

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark Using DataframeBABB0.75541.51082.26623.02163.7773.3241211423.3476742133.357334893

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark Using DataframeBABB0.7591.5182.2773.0363.7953.3400000003.3600000003.373333355

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - Calculate Pi Benchmark Using DataframeABBB0.78081.56162.34243.12323.9043.443.443.47

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Calculate Pi BenchmarkBBAB112233445550.0150.0550.31

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Calculate Pi Benchmark Using DataframeBBBA0.7651.532.2953.063.8253.3800000003.3900000003.400212784

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - Calculate Pi Benchmark Using DataframeBBBA0.75171.50342.25513.00683.75853.3300000003.3400000003.341043375

120 Results Shown

Apache Spark:
  20000000 - 1000 - Inner Join Test Time
  40000000 - 500 - Inner Join Test Time
  40000000 - 2000 - Repartition Test Time
  20000000 - 100 - Inner Join Test Time
  40000000 - 500 - Repartition Test Time
  10000000 - 500 - Broadcast Inner Join Test Time
  20000000 - 2000 - Broadcast Inner Join Test Time
  40000000 - 1000 - Broadcast Inner Join Test Time
  40000000 - 1000 - Repartition Test Time
  20000000 - 500 - Inner Join Test Time
  40000000 - 100 - Repartition Test Time
  20000000 - 2000 - Repartition Test Time
  20000000 - 100 - Broadcast Inner Join Test Time
  20000000 - 2000 - Inner Join Test Time
  20000000 - 1000 - Broadcast Inner Join Test Time
  20000000 - 1000 - Repartition Test Time
  40000000 - 500 - Broadcast Inner Join Test Time
  40000000 - 2000 - Broadcast Inner Join Test Time
  20000000 - 100 - Repartition Test Time
  40000000 - 2000 - Inner Join Test Time
  20000000 - 500 - Broadcast Inner Join Test Time
  20000000 - 500 - Repartition Test Time
  10000000 - 500 - Inner Join Test Time
  40000000 - 1000 - Inner Join Test Time
  40000000 - 2000 - SHA-512 Benchmark Time
  40000000 - 500 - SHA-512 Benchmark Time
  10000000 - 100 - Broadcast Inner Join Test Time
  40000000 - 100 - Inner Join Test Time
  40000000 - 100 - Broadcast Inner Join Test Time
  40000000 - 100 - SHA-512 Benchmark Time
  20000000 - 100 - SHA-512 Benchmark Time
  10000000 - 100 - Inner Join Test Time
  20000000 - 1000 - SHA-512 Benchmark Time
  40000000 - 1000 - SHA-512 Benchmark Time
  10000000 - 2000 - Repartition Test Time
  10000000 - 100 - Repartition Test Time
  10000000 - 2000 - Broadcast Inner Join Test Time
  20000000 - 500 - SHA-512 Benchmark Time
  10000000 - 500 - Repartition Test Time
  10000000 - 2000 - Inner Join Test Time
  10000000 - 1000 - Broadcast Inner Join Test Time
  20000000 - 2000 - SHA-512 Benchmark Time
  10000000 - 1000 - Repartition Test Time
  10000000 - 1000 - Inner Join Test Time
  40000000 - 2000 - Group By Test Time
  1000000 - 1000 - Broadcast Inner Join Test Time
  10000000 - 2000 - SHA-512 Benchmark Time
  10000000 - 1000 - SHA-512 Benchmark Time
  10000000 - 500 - SHA-512 Benchmark Time
  40000000 - 500 - Group By Test Time
  40000000 - 100 - Group By Test Time
  40000000 - 1000 - Group By Test Time
  10000000 - 100 - SHA-512 Benchmark Time
  1000000 - 500 - Broadcast Inner Join Test Time
SVT-AV1:
  Preset 8 - Bosphorus 4K
  Preset 10 - Bosphorus 4K
  Preset 12 - Bosphorus 4K
Apache Spark:
  20000000 - 1000 - Group By Test Time
  20000000 - 2000 - Group By Test Time
  1000000 - 500 - Inner Join Test Time
  1000000 - 1000 - Inner Join Test Time
  20000000 - 100 - Group By Test Time
  1000000 - 1000 - Repartition Test Time
  20000000 - 500 - Group By Test Time
  1000000 - 500 - Repartition Test Time
  10000000 - 1000 - Group By Test Time
  10000000 - 2000 - Group By Test Time
  1000000 - 2000 - Inner Join Test Time
  10000000 - 100 - Group By Test Time
  1000000 - 2000 - Repartition Test Time
  10000000 - 500 - Group By Test Time
  1000000 - 100 - Inner Join Test Time
  1000000 - 100 - Repartition Test Time
  1000000 - 2000 - Broadcast Inner Join Test Time
  1000000 - 100 - Broadcast Inner Join Test Time
SVT-AV1:
  Preset 10 - Bosphorus 1080p
  Preset 12 - Bosphorus 1080p
Apache Spark:
  1000000 - 2000 - SHA-512 Benchmark Time
  1000000 - 1000 - SHA-512 Benchmark Time
  1000000 - 1000 - Group By Test Time
  1000000 - 500 - SHA-512 Benchmark Time
  1000000 - 100 - Group By Test Time
SVT-AV1
Apache Spark:
  1000000 - 500 - Group By Test Time
  1000000 - 100 - SHA-512 Benchmark Time
  1000000 - 2000 - Group By Test Time
  40000000 - 500 - Calculate Pi Benchmark Using Dataframe
  1000000 - 100 - Calculate Pi Benchmark
  1000000 - 500 - Calculate Pi Benchmark Using Dataframe
  10000000 - 500 - Calculate Pi Benchmark Using Dataframe
  20000000 - 100 - Calculate Pi Benchmark Using Dataframe
  40000000 - 2000 - Calculate Pi Benchmark
  10000000 - 1000 - Calculate Pi Benchmark
  40000000 - 2000 - Calculate Pi Benchmark Using Dataframe
  20000000 - 1000 - Calculate Pi Benchmark Using Dataframe
  20000000 - 500 - Calculate Pi Benchmark Using Dataframe
SVT-AV1
Apache Spark:
  20000000 - 500 - Calculate Pi Benchmark
  40000000 - 1000 - Calculate Pi Benchmark
  40000000 - 1000 - Calculate Pi Benchmark Using Dataframe
  10000000 - 500 - Calculate Pi Benchmark
  1000000 - 500 - Calculate Pi Benchmark
  10000000 - 2000 - Calculate Pi Benchmark Using Dataframe
  40000000 - 100 - Calculate Pi Benchmark
  10000000 - 100 - Calculate Pi Benchmark Using Dataframe
  20000000 - 2000 - Calculate Pi Benchmark Using Dataframe
  1000000 - 2000 - Calculate Pi Benchmark
  20000000 - 2000 - Calculate Pi Benchmark
  1000000 - 1000 - Calculate Pi Benchmark
  10000000 - 2000 - Calculate Pi Benchmark
  40000000 - 500 - Calculate Pi Benchmark
SVT-AV1
Apache Spark:
  20000000 - 1000 - Calculate Pi Benchmark
  10000000 - 100 - Calculate Pi Benchmark
  1000000 - 2000 - Calculate Pi Benchmark Using Dataframe
  1000000 - 100 - Calculate Pi Benchmark Using Dataframe
  10000000 - 1000 - Calculate Pi Benchmark Using Dataframe
  20000000 - 100 - Calculate Pi Benchmark
  1000000 - 1000 - Calculate Pi Benchmark Using Dataframe
  40000000 - 100 - Calculate Pi Benchmark Using Dataframe