tr5 spark svt av1

Benchmarks for a future article.

Compare your own system(s) to this result file with the Phoronix Test Suite by running the command: phoronix-test-suite benchmark 2208075-NE-TR5SPARKS30
Jump To Table - Results

View

Do Not Show Noisy Results
Do Not Show Results With Incomplete Data
Do Not Show Results With Little Change/Spread
List Notable Results

Statistics

Show Overall Harmonic Mean(s)
Show Overall Geometric Mean
Show Wins / Losses Counts (Pie Chart)
Normalize Results
Remove Outliers Before Calculating Averages

Graph Settings

Force Line Graphs Where Applicable
Convert To Scalar Where Applicable
Prefer Vertical Bar Graphs

Multi-Way Comparison

Condense Multi-Option Tests Into Single Result Graphs

Table

Show Detailed System Result Table

Run Management

Highlight
Result
Hide
Result
Result
Identifier
View Logs
Performance Per
Dollar
Date
Run
  Test
  Duration
A
August 06 2022
  4 Hours, 9 Minutes
B
August 06 2022
  3 Hours, 59 Minutes
BB
August 07 2022
  5 Hours, 12 Minutes
Invert Hiding All Results Option
  4 Hours, 27 Minutes

Only show results where is faster than
Only show results matching title/arguments (delimit multiple options with a comma):
Do not show results matching title/arguments (delimit multiple options with a comma):


tr5 spark svt av1OpenBenchmarking.orgPhoronix Test SuiteAMD Ryzen Threadripper PRO 5965WX 24-Cores @ 3.80GHz (24 Cores / 48 Threads)ASUS Pro WS WRX80E-SAGE SE WIFI (1003 BIOS)AMD Starship/Matisse64GB1000GB Western Digital WDS100T1X0E-00AFY0ASUS NVIDIA NV106 2GBAMD Starship/MatisseVA24312 x Intel 10G X550T + Intel Wi-Fi 6 AX200Ubuntu 22.045.19.0-051900rc7-generic (x86_64)GNOME Shell 42.2X Server 1.21.1.3 + Waylandnouveau4.3 Mesa 22.0.11.2.204GCC 11.2.0ext41920x1080ProcessorMotherboardChipsetMemoryDiskGraphicsAudioMonitorNetworkOSKernelDesktopDisplay ServerDisplay DriverOpenGLVulkanCompilerFile-SystemScreen ResolutionTr5 Spark Svt Av1 BenchmarksSystem Logs- Transparent Huge Pages: madvise- --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-11-gBFGDP/gcc-11-11.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-gBFGDP/gcc-11-11.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v - Scaling Governor: acpi-cpufreq schedutil (Boost: Enabled) - CPU Microcode: 0xa008203- OpenJDK Runtime Environment (build 11.0.15+10-Ubuntu-0ubuntu0.22.04.1)- Python 3.10.4- itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling + srbds: Not affected + tsx_async_abort: Not affected

Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - Inner Join Test TimeBBBA61218243024.0614.2413.13

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - Inner Join Test TimeBBBA112233445547.2926.1127.32

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - Repartition Test TimeBBBA91827364541.2122.9523.65

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Inner Join Test TimeBBBA61218243024.1713.5015.12

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - Repartition Test TimeBBBA102030405042.3023.8424.33

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - Broadcast Inner Join Test TimeBBBA369121512.3333898347.06000000011.241856909

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - Broadcast Inner Join Test TimeBBBA61218243023.5113.4813.96

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - Broadcast Inner Join Test TimeBBBA102030405045.7828.4426.27

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - Repartition Test TimeBBBA102030405042.3424.3224.36

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - Inner Join Test TimeBBBA61218243025.4214.6317.64

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - Repartition Test TimeBBBA102030405042.4724.4526.70

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - Repartition Test TimeBBBA51015202521.3812.3813.06

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Broadcast Inner Join Test TimeBBBA61218243023.6813.7416.01

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - Inner Join Test TimeBBBA61218243024.2314.1714.69

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - Broadcast Inner Join Test TimeBBBA51015202522.5813.8013.23

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - Repartition Test TimeBBBA51015202521.0312.3912.37

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - Broadcast Inner Join Test TimeBBBA102030405045.4226.8127.60

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - Broadcast Inner Join Test TimeBBBA102030405042.8225.3926.03

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Repartition Test TimeBBBA51015202521.3212.6514.05

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - Inner Join Test TimeBBBA102030405044.6226.4726.80

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - Broadcast Inner Join Test TimeBBBA61218243024.3214.4717.42

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - Repartition Test TimeBBBA51015202521.9213.0815.75

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - Inner Join Test TimeBBBA369121512.9303795147.76000000011.790956912

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - Inner Join Test TimeBBBA102030405044.5626.7927.76

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - SHA-512 Benchmark TimeBBBA142842567060.8337.2836.70

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - SHA-512 Benchmark TimeBBBA142842567063.3838.4940.74

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Broadcast Inner Join Test TimeBBBA369121512.2649143097.48000000011.721111931

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - Inner Join Test TimeBBBA102030405044.8027.5628.25

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - Broadcast Inner Join Test TimeBBBA102030405046.0328.3328.72

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - SHA-512 Benchmark TimeBBBA142842567061.5737.9938.82

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - SHA-512 Benchmark TimeBBBA81624324032.9820.3724.18

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Inner Join Test TimeBBBA369121512.5180762037.76000000011.428134919

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - SHA-512 Benchmark TimeBBBA81624324032.7520.5420.46

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - SHA-512 Benchmark TimeBBBA132639526560.1337.5837.70

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Repartition Test TimeBBBA369121511.3834970017.1200000007.850000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Repartition Test TimeBBBA369121511.1438079546.99000000010.128496839

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Broadcast Inner Join Test TimeBBBA369121512.658960587.950000008.39000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - SHA-512 Benchmark TimeBBBA81624324032.6220.5324.41

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - Repartition Test TimeBBBA369121511.1304972417.0300000009.790000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Inner Join Test TimeBBBA369121513.1486852878.3100000008.590000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - Broadcast Inner Join Test TimeBBBA369121511.5859537677.4100000007.850000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - SHA-512 Benchmark TimeBBBA81624324032.5520.9821.27

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - Repartition Test TimeBBBA369121510.6679781286.9200000007.560000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - Inner Join Test TimeBBBA369121512.2070165218.0300000008.210000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - Group By Test TimeBBBA81624324033.3922.2623.46

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Broadcast Inner Join Test TimeBBBA0.55011.10021.65032.20042.75052.4448487751.6391141811.969098985

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - SHA-512 Benchmark TimeBBBA4812162017.1111.5912.40

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - SHA-512 Benchmark TimeBBBA4812162017.2711.7112.18

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - SHA-512 Benchmark TimeBBBA4812162017.6112.0415.41

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - Group By Test TimeBBBA81624324033.3422.8724.77

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - Group By Test TimeBBBA81624324033.1122.8426.58

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - Group By Test TimeBBBA81624324032.5922.7824.03

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - SHA-512 Benchmark TimeBBBA4812162017.0711.9415.13

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Broadcast Inner Join Test TimeBBBA0.37510.75021.12531.50041.87551.6671049161.2600000001.471185098

SVT-AV1

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 8 - Input: Bosphorus 4KBBBA153045607551.3766.7359.251. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 10 - Input: Bosphorus 4KBBBA2040608010086.16111.01102.121. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 12 - Input: Bosphorus 4KBBBA306090120150108.38139.51129.141. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - Group By Test TimeBBBA369121511.5031995658.9500000009.230000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - Group By Test TimeBBBA369121511.8503749149.3500000009.912154914

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Inner Join Test TimeBBBA0.51891.03781.55672.07562.59452.3062980591.8400000001.959952423

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Inner Join Test TimeBBBA0.57481.14961.72442.29922.8742.5546306392.0400000002.168718063

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Group By Test TimeBBBA369121511.6653610279.38000000010.857588257

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Repartition Test TimeBBBA0.57131.14261.71392.28522.85652.5389242662.0500000002.140000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - Group By Test TimeBBBA369121511.7599913199.51000000010.595039675

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Repartition Test TimeBBBA0.53031.06061.59092.12122.65152.3568674441.9100000001.970000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - Group By Test TimeBBBA2468107.926.556.93

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Group By Test TimeBBBA2468108.437.037.36

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Inner Join Test TimeBBBA0.6931.3862.0792.7723.4653.0800000002.5767158782.850000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Group By Test TimeBBBA2468108.0596100096.8100000007.240000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Repartition Test TimeBBBA0.64341.28681.93022.57363.2172.8597524152.4200000002.457157809

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - Group By Test TimeBBBA2468107.7400000006.5700000007.156971652

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Inner Join Test TimeBBBA0.4640.9281.3921.8562.322.0622091711.7600000001.886124058

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Repartition Test TimeBBBA0.49150.9831.47451.9662.45752.1846570491.8900000001.890000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Broadcast Inner Join Test TimeBBBA0.60081.20161.80242.40323.0042.672.492.32

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Broadcast Inner Join Test TimeBBBA0.31830.63660.95491.27321.59151.414877911.230000001.33000000

SVT-AV1

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 10 - Input: Bosphorus 1080pBBBA70140210280350298.10341.35309.011. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 12 - Input: Bosphorus 1080pBBBA100200300400500411.29461.47418.671. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - SHA-512 Benchmark TimeBBBA0.97971.95942.93913.91884.89854.3543684133.9000000004.160000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - SHA-512 Benchmark TimeBBBA0.92821.85642.78463.71284.6414.1251416193.7053393093.970000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Group By Test TimeBBBA1.0212.0423.0634.0845.1054.5378299994.1500000004.310000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - SHA-512 Benchmark TimeBBBA0.85731.71462.57193.42924.28653.774602653.510000003.81000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Group By Test TimeBBBA0.99191.98382.97573.96764.95954.4085646234.0800000004.179243959

SVT-AV1

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 8 - Input: Bosphorus 1080pBBBA4080120160200149.40161.39152.981. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Group By Test TimeBBBA0.96021.92042.88063.84084.8014.2677366034.0100000004.028182532

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - SHA-512 Benchmark TimeBBBA0.84911.69822.54733.39644.24553.7525238303.5500000003.773601794

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Group By Test TimeBBBA1.12682.25363.38044.50725.6345.0080395194.7400000004.718211324

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - Calculate Pi Benchmark Using DataframeBBBA0.78281.56562.34843.13123.9143.3900000003.3500000003.479319347

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Calculate Pi BenchmarkBBBA112233445550.7949.0650.75

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Calculate Pi Benchmark Using DataframeBBBA0.77921.55842.33763.11683.8963.4632132243.4100000003.347404557

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - Calculate Pi Benchmark Using DataframeBBBA0.7831.5662.3493.1323.9153.4039372613.4800000003.370000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Calculate Pi Benchmark Using DataframeBBBA0.7651.532.2953.063.8253.3313364023.3100000003.400000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - Calculate Pi BenchmarkBBBA122436486051.4150.0750.27

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - Calculate Pi BenchmarkBBBA112233445550.7249.5349.94

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - Calculate Pi Benchmark Using DataframeBBBA0.7741.5482.3223.0963.873.443.443.36

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - Calculate Pi Benchmark Using DataframeBBBA0.7741.5482.3223.0963.873.443.363.42

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - Calculate Pi Benchmark Using DataframeBBBA0.77631.55262.32893.10523.88153.383.373.45

SVT-AV1

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 4 - Input: Bosphorus 1080pBBBA2468107.3877.4847.3121. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - Calculate Pi BenchmarkBBBA112233445550.8549.7050.52

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - Calculate Pi BenchmarkBBBA122436486051.0450.0449.90

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - Calculate Pi Benchmark Using DataframeBBBA0.7741.5482.3223.0963.873.443.373.39

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - Calculate Pi BenchmarkBBBA112233445550.0450.7349.78

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Calculate Pi BenchmarkBBBA112233445549.8149.2950.22

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Calculate Pi Benchmark Using DataframeBBBA0.76951.5392.30853.0783.84753.383.423.36

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - Calculate Pi BenchmarkBBBA112233445550.0449.7950.61

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Calculate Pi Benchmark Using DataframeBBBA0.76731.53462.30193.06923.83653.3903942883.4100000003.360000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - Calculate Pi Benchmark Using DataframeBBBA0.7741.5482.3223.0963.873.403.443.39

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Calculate Pi BenchmarkBBBA112233445549.7450.4750.43

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - Calculate Pi BenchmarkBBBA112233445549.7549.7150.39

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Calculate Pi BenchmarkBBBA112233445549.3249.4649.99

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Calculate Pi BenchmarkBBBA112233445550.0750.1550.71

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - Calculate Pi BenchmarkBBBA112233445550.1249.5249.74

SVT-AV1

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 4 - Input: Bosphorus 4KBBBA0.64551.2911.93652.5823.22752.8362.8562.8691. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - Calculate Pi BenchmarkBBBA112233445550.2249.9450.50

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Calculate Pi BenchmarkBBBA112233445550.3450.5850.07

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark Using DataframeBBBA0.75541.51082.26623.02163.7773.3573348933.3241211423.347674213

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark Using DataframeBBBA0.7591.5182.2773.0363.7953.3733333553.3400000003.360000000

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - Calculate Pi Benchmark Using DataframeBBBA0.78081.56162.34243.12323.9043.443.473.44

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Calculate Pi BenchmarkBBBA112233445550.0150.3150.05

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Calculate Pi Benchmark Using DataframeBBBA0.7651.532.2953.063.8253.3800000003.3900000003.400212784

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - Calculate Pi Benchmark Using DataframeBBBA0.75171.50342.25513.00683.75853.3300000003.3400000003.341043375

120 Results Shown

Apache Spark:
  20000000 - 1000 - Inner Join Test Time
  40000000 - 500 - Inner Join Test Time
  40000000 - 2000 - Repartition Test Time
  20000000 - 100 - Inner Join Test Time
  40000000 - 500 - Repartition Test Time
  10000000 - 500 - Broadcast Inner Join Test Time
  20000000 - 2000 - Broadcast Inner Join Test Time
  40000000 - 1000 - Broadcast Inner Join Test Time
  40000000 - 1000 - Repartition Test Time
  20000000 - 500 - Inner Join Test Time
  40000000 - 100 - Repartition Test Time
  20000000 - 2000 - Repartition Test Time
  20000000 - 100 - Broadcast Inner Join Test Time
  20000000 - 2000 - Inner Join Test Time
  20000000 - 1000 - Broadcast Inner Join Test Time
  20000000 - 1000 - Repartition Test Time
  40000000 - 500 - Broadcast Inner Join Test Time
  40000000 - 2000 - Broadcast Inner Join Test Time
  20000000 - 100 - Repartition Test Time
  40000000 - 2000 - Inner Join Test Time
  20000000 - 500 - Broadcast Inner Join Test Time
  20000000 - 500 - Repartition Test Time
  10000000 - 500 - Inner Join Test Time
  40000000 - 1000 - Inner Join Test Time
  40000000 - 2000 - SHA-512 Benchmark Time
  40000000 - 500 - SHA-512 Benchmark Time
  10000000 - 100 - Broadcast Inner Join Test Time
  40000000 - 100 - Inner Join Test Time
  40000000 - 100 - Broadcast Inner Join Test Time
  40000000 - 100 - SHA-512 Benchmark Time
  20000000 - 100 - SHA-512 Benchmark Time
  10000000 - 100 - Inner Join Test Time
  20000000 - 1000 - SHA-512 Benchmark Time
  40000000 - 1000 - SHA-512 Benchmark Time
  10000000 - 2000 - Repartition Test Time
  10000000 - 100 - Repartition Test Time
  10000000 - 2000 - Broadcast Inner Join Test Time
  20000000 - 500 - SHA-512 Benchmark Time
  10000000 - 500 - Repartition Test Time
  10000000 - 2000 - Inner Join Test Time
  10000000 - 1000 - Broadcast Inner Join Test Time
  20000000 - 2000 - SHA-512 Benchmark Time
  10000000 - 1000 - Repartition Test Time
  10000000 - 1000 - Inner Join Test Time
  40000000 - 2000 - Group By Test Time
  1000000 - 1000 - Broadcast Inner Join Test Time
  10000000 - 2000 - SHA-512 Benchmark Time
  10000000 - 1000 - SHA-512 Benchmark Time
  10000000 - 500 - SHA-512 Benchmark Time
  40000000 - 500 - Group By Test Time
  40000000 - 100 - Group By Test Time
  40000000 - 1000 - Group By Test Time
  10000000 - 100 - SHA-512 Benchmark Time
  1000000 - 500 - Broadcast Inner Join Test Time
SVT-AV1:
  Preset 8 - Bosphorus 4K
  Preset 10 - Bosphorus 4K
  Preset 12 - Bosphorus 4K
Apache Spark:
  20000000 - 1000 - Group By Test Time
  20000000 - 2000 - Group By Test Time
  1000000 - 500 - Inner Join Test Time
  1000000 - 1000 - Inner Join Test Time
  20000000 - 100 - Group By Test Time
  1000000 - 1000 - Repartition Test Time
  20000000 - 500 - Group By Test Time
  1000000 - 500 - Repartition Test Time
  10000000 - 1000 - Group By Test Time
  10000000 - 2000 - Group By Test Time
  1000000 - 2000 - Inner Join Test Time
  10000000 - 100 - Group By Test Time
  1000000 - 2000 - Repartition Test Time
  10000000 - 500 - Group By Test Time
  1000000 - 100 - Inner Join Test Time
  1000000 - 100 - Repartition Test Time
  1000000 - 2000 - Broadcast Inner Join Test Time
  1000000 - 100 - Broadcast Inner Join Test Time
SVT-AV1:
  Preset 10 - Bosphorus 1080p
  Preset 12 - Bosphorus 1080p
Apache Spark:
  1000000 - 2000 - SHA-512 Benchmark Time
  1000000 - 1000 - SHA-512 Benchmark Time
  1000000 - 1000 - Group By Test Time
  1000000 - 500 - SHA-512 Benchmark Time
  1000000 - 100 - Group By Test Time
SVT-AV1
Apache Spark:
  1000000 - 500 - Group By Test Time
  1000000 - 100 - SHA-512 Benchmark Time
  1000000 - 2000 - Group By Test Time
  40000000 - 500 - Calculate Pi Benchmark Using Dataframe
  1000000 - 100 - Calculate Pi Benchmark
  1000000 - 500 - Calculate Pi Benchmark Using Dataframe
  10000000 - 500 - Calculate Pi Benchmark Using Dataframe
  20000000 - 100 - Calculate Pi Benchmark Using Dataframe
  40000000 - 2000 - Calculate Pi Benchmark
  10000000 - 1000 - Calculate Pi Benchmark
  40000000 - 2000 - Calculate Pi Benchmark Using Dataframe
  20000000 - 1000 - Calculate Pi Benchmark Using Dataframe
  20000000 - 500 - Calculate Pi Benchmark Using Dataframe
SVT-AV1
Apache Spark:
  20000000 - 500 - Calculate Pi Benchmark
  40000000 - 1000 - Calculate Pi Benchmark
  40000000 - 1000 - Calculate Pi Benchmark Using Dataframe
  10000000 - 500 - Calculate Pi Benchmark
  1000000 - 500 - Calculate Pi Benchmark
  10000000 - 2000 - Calculate Pi Benchmark Using Dataframe
  40000000 - 100 - Calculate Pi Benchmark
  10000000 - 100 - Calculate Pi Benchmark Using Dataframe
  20000000 - 2000 - Calculate Pi Benchmark Using Dataframe
  1000000 - 2000 - Calculate Pi Benchmark
  20000000 - 2000 - Calculate Pi Benchmark
  1000000 - 1000 - Calculate Pi Benchmark
  10000000 - 2000 - Calculate Pi Benchmark
  40000000 - 500 - Calculate Pi Benchmark
SVT-AV1
Apache Spark:
  20000000 - 1000 - Calculate Pi Benchmark
  10000000 - 100 - Calculate Pi Benchmark
  1000000 - 2000 - Calculate Pi Benchmark Using Dataframe
  1000000 - 100 - Calculate Pi Benchmark Using Dataframe
  10000000 - 1000 - Calculate Pi Benchmark Using Dataframe
  20000000 - 100 - Calculate Pi Benchmark
  1000000 - 1000 - Calculate Pi Benchmark Using Dataframe
  40000000 - 100 - Calculate Pi Benchmark Using Dataframe