spark svt

AMD EPYC 7551 32-Core testing with a GIGABYTE MZ31-AR0-00 v01010101 (F10 BIOS) and ASPEED on Debian 11 via the Phoronix Test Suite.

Compare your own system(s) to this result file with the Phoronix Test Suite by running the command: phoronix-test-suite benchmark 2208045-HA-SPARKSVT093
Jump To Table - Results

View

Do Not Show Noisy Results
Do Not Show Results With Incomplete Data
Do Not Show Results With Little Change/Spread
List Notable Results

Statistics

Show Overall Harmonic Mean(s)
Show Overall Geometric Mean
Show Wins / Losses Counts (Pie Chart)
Normalize Results
Remove Outliers Before Calculating Averages

Graph Settings

Force Line Graphs Where Applicable
Convert To Scalar Where Applicable
Prefer Vertical Bar Graphs

Multi-Way Comparison

Condense Multi-Option Tests Into Single Result Graphs

Table

Show Detailed System Result Table

Run Management

Highlight
Result
Hide
Result
Result
Identifier
View Logs
Performance Per
Dollar
Date
Run
  Test
  Duration
A
August 04 2022
  6 Hours, 51 Minutes
B
August 04 2022
  6 Hours, 48 Minutes
C
August 04 2022
  6 Hours, 44 Minutes
D
August 04 2022
  6 Hours, 47 Minutes
Invert Hiding All Results Option
  6 Hours, 48 Minutes

Only show results where is faster than
Only show results matching title/arguments (delimit multiple options with a comma):
Do not show results matching title/arguments (delimit multiple options with a comma):


spark svtOpenBenchmarking.orgPhoronix Test SuiteAMD EPYC 7551 32-Core @ 2.00GHz (32 Cores / 64 Threads)GIGABYTE MZ31-AR0-00 v01010101 (F10 BIOS)AMD 17h8 x 4 GB DDR4-2667MT/s 9ASF51272PZ-2G6E1Samsung SSD 960 EVO 500GB + 31GB SanDisk 3.2Gen1ASPEEDRealtek RTL8111/8168/8411 + 2 x Broadcom NetXtreme II BCM57810 10Debian 115.10.0-9-amd64 (x86_64)GCC 10.2.1 20210110ext41024x768ProcessorMotherboardChipsetMemoryDiskGraphicsNetworkOSKernelCompilerFile-SystemScreen ResolutionSpark Svt BenchmarksSystem Logs- --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-mutex --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-10-Km9U7s/gcc-10-10.2.1/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-10-Km9U7s/gcc-10-10.2.1/debian/tmp-gcn/usr,hsa --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v - Scaling Governor: acpi-cpufreq schedutil (Boost: Enabled) - CPU Microcode: 0x8001227- OpenJDK Runtime Environment (build 11.0.12+7-post-Debian-2)- Python 3.9.2- itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl and seccomp + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Full AMD retpoline IBPB: conditional STIBP: disabled RSB filling + srbds: Not affected + tsx_async_abort: Not affected

ABCDResult OverviewPhoronix Test Suite100%126%153%179%205%Apache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkSVT-AV1Apache SparkApache SparkApache SparkSVT-AV1Apache SparkApache SparkApache SparkApache SparkApache SparkApache SparkSVT-AV1Apache SparkApache SparkApache SparkSVT-AV1Apache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkSVT-AV1Apache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkSVT-AV1Apache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkSVT-AV1Apache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkSVT-AV1Apache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache SparkApache Spark1000000 - 100 - R.T.T1000000 - 1000 - R.T.T1000000 - 2000 - B.I.J.T.T1000000 - 100 - B.I.J.T.T1000000 - 500 - B.I.J.T.T1000000 - 500 - R.T.T1000000 - 100 - I.J.T.T10000000 - 100 - R.T.T1000000 - 1000 - B.I.J.T.T40000000 - 100 - S.5.B.T10000000 - 500 - Group By Test Time20000000 - 100 - S.5.B.T1000000 - 500 - I.J.T.T1000000 - 1000 - I.J.T.T10000000 - 2000 - B.I.J.T.T1000000 - 2000 - R.T.T20000000 - 1000 - Group By Test Time20000000 - 500 - R.T.TPreset 10 - Bosphorus 4K10000000 - 500 - R.T.T10000000 - 1000 - Group By Test Time10000000 - 100 - Group By Test TimePreset 8 - Bosphorus 4K20000000 - 100 - R.T.T1000000 - 1000 - S.5.B.T10000000 - 2000 - I.J.T.T10000000 - 2000 - Group By Test Time10000000 - 1000 - B.I.J.T.T10000000 - 500 - I.J.T.TPreset 10 - Bosphorus 1080p20000000 - 1000 - S.5.B.T20000000 - 2000 - R.T.T40000000 - 2000 - I.J.T.TPreset 12 - Bosphorus 1080p10000000 - 100 - I.J.T.T20000000 - 100 - I.J.T.T20000000 - 100 - B.I.J.T.T10000000 - 100 - S.5.B.T40000000 - 1000 - S.5.B.T10000000 - 500 - B.I.J.T.T20000000 - 100 - Group By Test Time1000000 - 2000 - I.J.T.T1000000 - 500 - S.5.B.TPreset 12 - Bosphorus 4K10000000 - 1000 - R.T.T40000000 - 100 - R.T.T40000000 - 100 - B.I.J.T.T10000000 - 2000 - S.5.B.T1000000 - 2000 - S.5.B.T40000000 - 500 - S.5.B.T20000000 - 500 - Group By Test Time20000000 - 1000 - R.T.T20000000 - 500 - B.I.J.T.T40000000 - 1000 - Group By Test Time10000000 - 100 - B.I.J.T.T40000000 - 100 - Group By Test Time10000000 - 1000 - I.J.T.T20000000 - 500 - S.5.B.T40000000 - 2000 - R.T.TPreset 8 - Bosphorus 1080p1000000 - 1000 - Group By Test Time40000000 - 2000 - S.5.B.T40000000 - 2000 - Group By Test Time20000000 - 500 - I.J.T.T1000000 - 2000 - Group By Test Time1000000 - 1000 - C.P.B.U.D10000000 - 2000 - R.T.T20000000 - 2000 - S.5.B.T1000000 - 500 - Group By Test Time1000000 - 100 - S.5.B.T10000000 - 1000 - S.5.B.T1000000 - 100 - Group By Test Time10000000 - 500 - C.P.B.U.DPreset 4 - Bosphorus 4K1000000 - 500 - C.P.B.U.D40000000 - 2000 - C.P.B.U.D20000000 - 500 - C.P.B.U.D40000000 - 2000 - B.I.J.T.T1000000 - 2000 - C.P.B.U.D40000000 - 500 - I.J.T.T20000000 - 2000 - I.J.T.T10000000 - 1000 - C.P.B.U.D20000000 - 2000 - Group By Test Time1000000 - 100 - C.P.B.U.D40000000 - 1000 - B.I.J.T.T40000000 - 500 - Group By Test Time40000000 - 1000 - C.P.B.U.D20000000 - 2000 - B.I.J.T.T40000000 - 500 - R.T.TPreset 4 - Bosphorus 1080p10000000 - 2000 - C.P.B.U.D40000000 - 100 - I.J.T.T40000000 - 1000 - I.J.T.T20000000 - 1000 - I.J.T.T20000000 - 1000 - C.P.B.U.D20000000 - 1000 - B.I.J.T.T40000000 - 500 - C.P.B.U.D40000000 - 100 - C.P.B.U.D40000000 - 500 - B.I.J.T.T20000000 - 2000 - C.P.B.U.D40000000 - 1000 - R.T.T

spark svtspark: 1000000 - 100 - Repartition Test Timespark: 1000000 - 1000 - Repartition Test Timespark: 1000000 - 2000 - Broadcast Inner Join Test Timespark: 1000000 - 100 - Broadcast Inner Join Test Timespark: 1000000 - 500 - Broadcast Inner Join Test Timespark: 1000000 - 500 - Repartition Test Timespark: 1000000 - 100 - Inner Join Test Timespark: 10000000 - 100 - Repartition Test Timespark: 1000000 - 1000 - Broadcast Inner Join Test Timespark: 40000000 - 100 - SHA-512 Benchmark Timespark: 10000000 - 500 - Group By Test Timespark: 20000000 - 100 - SHA-512 Benchmark Timespark: 1000000 - 500 - Inner Join Test Timespark: 1000000 - 1000 - Inner Join Test Timespark: 10000000 - 2000 - Broadcast Inner Join Test Timespark: 1000000 - 2000 - Repartition Test Timespark: 20000000 - 1000 - Group By Test Timespark: 20000000 - 500 - Repartition Test Timesvt-av1: Preset 10 - Bosphorus 4Kspark: 10000000 - 500 - Repartition Test Timespark: 10000000 - 1000 - Group By Test Timespark: 10000000 - 100 - Group By Test Timesvt-av1: Preset 8 - Bosphorus 4Kspark: 20000000 - 100 - Repartition Test Timespark: 1000000 - 1000 - SHA-512 Benchmark Timespark: 10000000 - 2000 - Inner Join Test Timespark: 10000000 - 2000 - Group By Test Timespark: 10000000 - 1000 - Broadcast Inner Join Test Timespark: 10000000 - 500 - Inner Join Test Timesvt-av1: Preset 10 - Bosphorus 1080pspark: 20000000 - 1000 - SHA-512 Benchmark Timespark: 20000000 - 2000 - Repartition Test Timespark: 40000000 - 2000 - Inner Join Test Timesvt-av1: Preset 12 - Bosphorus 1080pspark: 10000000 - 100 - Inner Join Test Timespark: 20000000 - 100 - Inner Join Test Timespark: 20000000 - 100 - Broadcast Inner Join Test Timespark: 10000000 - 100 - SHA-512 Benchmark Timespark: 40000000 - 1000 - SHA-512 Benchmark Timespark: 10000000 - 500 - Broadcast Inner Join Test Timespark: 20000000 - 100 - Group By Test Timespark: 1000000 - 2000 - Inner Join Test Timespark: 1000000 - 500 - SHA-512 Benchmark Timesvt-av1: Preset 12 - Bosphorus 4Kspark: 10000000 - 1000 - Repartition Test Timespark: 40000000 - 100 - Repartition Test Timespark: 40000000 - 100 - Broadcast Inner Join Test Timespark: 10000000 - 2000 - SHA-512 Benchmark Timespark: 1000000 - 2000 - SHA-512 Benchmark Timespark: 40000000 - 500 - SHA-512 Benchmark Timespark: 20000000 - 500 - Group By Test Timespark: 20000000 - 1000 - Repartition Test Timespark: 20000000 - 500 - Broadcast Inner Join Test Timespark: 40000000 - 1000 - Group By Test Timespark: 10000000 - 100 - Broadcast Inner Join Test Timespark: 40000000 - 100 - Group By Test Timespark: 10000000 - 1000 - Inner Join Test Timespark: 20000000 - 500 - SHA-512 Benchmark Timespark: 40000000 - 2000 - Repartition Test Timesvt-av1: Preset 8 - Bosphorus 1080pspark: 1000000 - 1000 - Group By Test Timespark: 40000000 - 2000 - SHA-512 Benchmark Timespark: 40000000 - 2000 - Group By Test Timespark: 20000000 - 500 - Inner Join Test Timespark: 1000000 - 2000 - Group By Test Timespark: 1000000 - 1000 - Calculate Pi Benchmark Using Dataframespark: 10000000 - 2000 - Repartition Test Timespark: 20000000 - 2000 - SHA-512 Benchmark Timespark: 1000000 - 500 - Group By Test Timespark: 1000000 - 100 - SHA-512 Benchmark Timespark: 10000000 - 1000 - SHA-512 Benchmark Timespark: 1000000 - 100 - Group By Test Timespark: 10000000 - 500 - Calculate Pi Benchmark Using Dataframesvt-av1: Preset 4 - Bosphorus 4Kspark: 1000000 - 500 - Calculate Pi Benchmark Using Dataframespark: 40000000 - 2000 - Calculate Pi Benchmark Using Dataframespark: 20000000 - 500 - Calculate Pi Benchmark Using Dataframespark: 40000000 - 2000 - Broadcast Inner Join Test Timespark: 1000000 - 2000 - Calculate Pi Benchmark Using Dataframespark: 40000000 - 500 - Inner Join Test Timespark: 20000000 - 2000 - Inner Join Test Timespark: 10000000 - 1000 - Calculate Pi Benchmark Using Dataframespark: 20000000 - 2000 - Group By Test Timespark: 1000000 - 100 - Calculate Pi Benchmark Using Dataframespark: 40000000 - 1000 - Broadcast Inner Join Test Timespark: 40000000 - 500 - Group By Test Timespark: 40000000 - 1000 - Calculate Pi Benchmark Using Dataframespark: 20000000 - 2000 - Broadcast Inner Join Test Timespark: 40000000 - 500 - Repartition Test Timesvt-av1: Preset 4 - Bosphorus 1080pspark: 10000000 - 2000 - Calculate Pi Benchmark Using Dataframespark: 40000000 - 100 - Inner Join Test Timespark: 40000000 - 1000 - Inner Join Test Timespark: 20000000 - 1000 - Inner Join Test Timespark: 20000000 - 1000 - Calculate Pi Benchmark Using Dataframespark: 20000000 - 1000 - Broadcast Inner Join Test Timespark: 40000000 - 500 - Calculate Pi Benchmark Using Dataframespark: 40000000 - 100 - Calculate Pi Benchmark Using Dataframespark: 40000000 - 500 - Broadcast Inner Join Test Timespark: 20000000 - 2000 - Calculate Pi Benchmark Using Dataframespark: 40000000 - 1000 - Repartition Test Timespark: 10000000 - 500 - SHA-512 Benchmark Timespark: 10000000 - 100 - Calculate Pi Benchmark Using Dataframespark: 20000000 - 2000 - Calculate Pi Benchmarkspark: 20000000 - 1000 - Calculate Pi Benchmarkspark: 10000000 - 1000 - Calculate Pi Benchmarkspark: 1000000 - 100 - Calculate Pi Benchmarkspark: 1000000 - 2000 - Calculate Pi Benchmarkspark: 40000000 - 100 - Calculate Pi Benchmarkspark: 40000000 - 500 - Calculate Pi Benchmarkspark: 10000000 - 100 - Calculate Pi Benchmarkspark: 40000000 - 1000 - Calculate Pi Benchmarkspark: 40000000 - 2000 - Calculate Pi Benchmarkspark: 10000000 - 2000 - Calculate Pi Benchmarkspark: 20000000 - 100 - Calculate Pi Benchmark Using Dataframespark: 1000000 - 500 - Calculate Pi Benchmarkspark: 20000000 - 500 - Calculate Pi Benchmarkspark: 20000000 - 100 - Calculate Pi Benchmarkspark: 10000000 - 500 - Calculate Pi Benchmarkspark: 1000000 - 1000 - Calculate Pi BenchmarkABCD4.645.435.623.103.165.093.6512.313.0279.4815.3036.154.404.5314.716.1921.5719.1849.1715.1317.0914.5528.87623.167.7617.0819.0912.9712.56129.46238.6323.4337.30193.67213.5121.7619.7022.3871.6712.2821.055.748.5762.04215.3737.6139.4123.008.3163.0821.8222.4218.5045.8711.6742.5813.8737.1134.2961.7829.9963.5445.7019.96169719110.056.0813.8739.289.857.6222.569.246.240.9456.026.036.2938.326.5735.6623.026.1823.476.0537.3646.766.3321.6035.702.4326.1938.2237.9221.886.0420.176.076.0635.726.1035.8221.916.4080.4280.7280.8678.6878.8480.3980.6978.8080.9680.6180.746.0378.3480.5379.1679.6878.883.875.687.742.563.565.985.0116.724.0661.7319.5435.844.585.0112.977.1922.6223.4940.56613.7216.5015.1024.23520.087.4314.9517.9811.1312.17119.49541.1520.3042.99170.27811.7920.0019.1721.4363.0311.2320.295.877.5760.30613.6533.4335.0425.038.6165.0619.5723.2818.1243.7010.5642.5913.7637.9536.6957.43310.8168.3846.8819.4610.776.4014.2639.199.307.1622.148.936.141.0086.426.416.1739.546.2035.8622.886.2823.716.3636.6345.476.0422.2434.162.5395.9336.8537.5421.326.0919.476.266.0936.896.0135.2722.516.2779.8179.7380.0680.0979.7380.8881.2079.5080.7980.4680.676.0379.1179.9179.7379.4478.844.775.304.572.054.477.293.9715.713.2560.3815.0034.034.435.4711.825.7623.7822.4142.50612.6415.0615.1825.16921.418.7515.3818.4711.9514.15131.75135.6320.3141.55196.22913.1122.9021.9024.0763.3612.2918.536.527.7154.82214.8534.6837.2922.317.7166.5720.6422.8420.0841.4811.1238.9814.4040.0334.3362.24110.4265.0043.4220.9510.136.5413.4636.559.537.6921.859.256.220.9746.266.096.0739.506.2137.6522.156.5022.796.2636.1244.616.2421.4235.642.4926.1636.6836.4221.066.1119.456.185.9036.395.9234.7722.006.3178.6579.4479.2779.5678.7079.7180.4378.4479.9180.0179.915.9778.5480.0279.1979.2778.567.949.225.712.913.115.833.5216.343.2871.7715.5044.065.624.3311.756.9719.2319.2046.59115.2614.2017.5125.41519.577.8114.5516.3011.7113.23138.69436.3722.0340.05195.50613.3522.5920.3424.3965.1910.8118.985.757.8255.32414.3436.6936.8722.898.5670.3821.5924.9018.7942.1910.9340.3913.1936.6837.2360.9510.1563.3245.9620.1710.016.1113.2637.419.997.5521.029.566.560.9846.306.025.9140.696.3236.9423.346.3722.556.2635.5845.736.2622.3934.352.5296.1837.7036.8221.836.2719.776.046.1136.096.0834.9421.956.4379.0979.0479.8478.7978.5079.7280.0778.8980.3979.7780.066.0278.6479.7679.2579.2278.62OpenBenchmarking.org

Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Repartition Test TimeDCAB2468107.944.774.643.87

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Repartition Test TimeDBAC36912159.225.685.435.30

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Broadcast Inner Join Test TimeBDAC2468107.745.715.624.57

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Broadcast Inner Join Test TimeADBC0.69751.3952.09252.793.48753.102.912.562.05

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Broadcast Inner Join Test TimeCBAD1.00582.01163.01744.02325.0294.473.563.163.11

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Repartition Test TimeCBDA2468107.295.985.835.09

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Inner Join Test TimeBCAD1.12732.25463.38194.50925.63655.013.973.653.52

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Repartition Test TimeBDCA4812162016.7216.3415.7112.31

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Broadcast Inner Join Test TimeBDCA0.91351.8272.74053.6544.56754.063.283.253.02

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - SHA-512 Benchmark TimeADBC2040608010079.4871.7761.7360.38

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - Group By Test TimeBDAC51015202519.5415.5015.3015.00

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - SHA-512 Benchmark TimeDABC102030405044.0636.1535.8434.03

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Inner Join Test TimeDBCA1.26452.5293.79355.0586.32255.624.584.434.40

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Inner Join Test TimeCBAD1.23082.46163.69244.92326.1545.475.014.534.33

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Broadcast Inner Join Test TimeABCD4812162014.7112.9711.8211.75

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Repartition Test TimeBDAC2468107.196.976.195.76

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - Group By Test TimeCBAD61218243023.7822.6221.5719.23

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - Repartition Test TimeBCDA61218243023.4922.4119.2019.18

SVT-AV1

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 10 - Input: Bosphorus 4KBCDA112233445540.5742.5146.5949.171. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - Repartition Test TimeDABC4812162015.2615.1313.7212.64

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - Group By Test TimeABCD4812162017.0916.5015.0614.20

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Group By Test TimeDCBA4812162017.5115.1815.1014.55

SVT-AV1

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 8 - Input: Bosphorus 4KBCDA71421283524.2425.1725.4228.881. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Repartition Test TimeACBD61218243023.1621.4120.0819.57

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - SHA-512 Benchmark TimeCDAB2468108.757.817.767.43

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Inner Join Test TimeACBD4812162017.0815.3814.9514.55

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Group By Test TimeACBD51015202519.0918.4717.9816.30

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - Broadcast Inner Join Test TimeACDB369121512.9711.9511.7111.13

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - Inner Join Test TimeCDAB4812162014.1513.2312.5612.17

SVT-AV1

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 10 - Input: Bosphorus 1080pBACD306090120150119.50129.46131.75138.691. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - SHA-512 Benchmark TimeBADC91827364541.1538.6336.3735.63

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - Repartition Test TimeADCB61218243023.4322.0320.3120.30

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - Inner Join Test TimeBCDA102030405042.9941.5540.0537.30

SVT-AV1

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 12 - Input: Bosphorus 1080pBADC4080120160200170.28193.67195.51196.231. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Inner Join Test TimeADCB369121513.5113.3513.1111.79

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Inner Join Test TimeCDAB51015202522.9022.5921.7620.00

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Broadcast Inner Join Test TimeCDAB51015202521.9020.3419.7019.17

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - SHA-512 Benchmark TimeDCAB61218243024.3924.0722.3821.43

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - SHA-512 Benchmark TimeADCB163248648071.6765.1963.3663.03

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - Broadcast Inner Join Test TimeCABD369121512.2912.2811.2310.81

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Group By Test TimeABDC51015202521.0520.2918.9818.53

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Inner Join Test TimeCBDA2468106.525.875.755.74

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - SHA-512 Benchmark TimeADCB2468108.577.827.717.57

SVT-AV1

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 12 - Input: Bosphorus 4KCDBA142842567054.8255.3260.3162.041. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - Repartition Test TimeACDB4812162015.3714.8514.3413.65

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - Repartition Test TimeADCB91827364537.6136.6934.6833.43

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - Broadcast Inner Join Test TimeACDB91827364539.4137.2936.8735.04

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - SHA-512 Benchmark TimeBADC61218243025.0323.0022.8922.31

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - SHA-512 Benchmark TimeBDAC2468108.618.568.317.71

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - SHA-512 Benchmark TimeDCBA163248648070.3866.5765.0663.08

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - Group By Test TimeADCB51015202521.8221.5920.6419.57

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - Repartition Test TimeDBCA61218243024.9023.2822.8422.42

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - Broadcast Inner Join Test TimeCDAB51015202520.0818.7918.5018.12

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - Group By Test TimeABDC102030405045.8743.7042.1941.48

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Broadcast Inner Join Test TimeACDB369121511.6711.1210.9310.56

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - Group By Test TimeBADC102030405042.5942.5840.3938.98

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - Inner Join Test TimeCABD4812162014.4013.8713.7613.19

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - SHA-512 Benchmark TimeCBAD91827364540.0337.9537.1136.68

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - Repartition Test TimeDBCA91827364537.2336.6934.3334.29

SVT-AV1

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 8 - Input: Bosphorus 1080pBDAC142842567057.4360.9561.7862.241. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Group By Test TimeBCDA369121510.8110.4210.159.99

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - SHA-512 Benchmark TimeBCAD153045607568.3865.0063.5463.32

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - Group By Test TimeBDAC112233445546.8845.9645.7043.42

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - Inner Join Test TimeCDAB51015202520.9520.1719.9619.46

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Group By Test TimeBCAD369121510.7710.1310.0510.01

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Calculate Pi Benchmark Using DataframeCBDA2468106.546.406.116.08

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Repartition Test TimeBACD4812162014.2613.8713.4613.26

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - SHA-512 Benchmark TimeABDC91827364539.2839.1937.4136.55

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Group By Test TimeDACB36912159.999.859.539.30

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - SHA-512 Benchmark TimeCADB2468107.697.627.557.16

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - SHA-512 Benchmark TimeABCD51015202522.5622.1421.8521.02

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Group By Test TimeDCAB36912159.569.259.248.93

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - Calculate Pi Benchmark Using DataframeDACB2468106.566.246.226.14

SVT-AV1

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 4 - Input: Bosphorus 4KACDB0.22680.45360.68040.90721.1340.9450.9740.9841.0081. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Calculate Pi Benchmark Using DataframeBDCA2468106.426.306.266.02

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - Calculate Pi Benchmark Using DataframeBCAD2468106.416.096.036.02

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - Calculate Pi Benchmark Using DataframeABCD2468106.296.176.075.91

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - Broadcast Inner Join Test TimeDBCA91827364540.6939.5439.5038.32

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark Using DataframeADCB2468106.576.326.216.20

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - Inner Join Test TimeCDBA91827364537.6536.9435.8635.66

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - Inner Join Test TimeDABC61218243023.3423.0222.8822.15

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - Calculate Pi Benchmark Using DataframeCDBA2468106.506.376.286.18

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - Group By Test TimeBACD61218243023.7123.4722.7922.55

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark Using DataframeBDCA2468106.366.266.266.05

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - Broadcast Inner Join Test TimeABCD91827364537.3636.6336.1235.58

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - Group By Test TimeADBC112233445546.7645.7345.4744.61

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - Calculate Pi Benchmark Using DataframeADCB2468106.336.266.246.04

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - Broadcast Inner Join Test TimeDBAC51015202522.3922.2421.6021.42

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - Repartition Test TimeACDB81624324035.7035.6434.3534.16

SVT-AV1

OpenBenchmarking.orgFrames Per Second, More Is BetterSVT-AV1 1.2Encoder Mode: Preset 4 - Input: Bosphorus 1080pACDB0.57131.14261.71392.28522.85652.4322.4922.5292.5391. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq

Apache Spark

This is a benchmark of Apache Spark with its PySpark interface. Apache Spark is an open-source unified analytics engine for large-scale data processing and dealing with big data. This test profile benchmars the Apache Spark in a single-system configuration using spark-submit. The test makes use of DIYBigData's pyspark-benchmark (https://github.com/DIYBigData/pyspark-benchmark/) for generating of test data and various Apache Spark operations. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Calculate Pi Benchmark Using DataframeADCB2468106.196.186.165.93

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - Inner Join Test TimeADBC91827364538.2237.7036.8536.68

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - Inner Join Test TimeABDC91827364537.9237.5436.8236.42

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - Inner Join Test TimeADBC51015202521.8821.8321.3221.06

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - Calculate Pi Benchmark Using DataframeDCBA2468106.276.116.096.04

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - Broadcast Inner Join Test TimeADBC51015202520.1719.7719.4719.45

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - Calculate Pi Benchmark Using DataframeBCAD2468106.266.186.076.04

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - Calculate Pi Benchmark Using DataframeDBAC2468106.116.096.065.90

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - Broadcast Inner Join Test TimeBCDA81624324036.8936.3936.0935.72

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - Calculate Pi Benchmark Using DataframeADBC2468106.106.086.015.92

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - Repartition Test TimeABDC81624324035.8235.2734.9434.77

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - SHA-512 Benchmark TimeBCDA51015202522.5122.0021.9521.91

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Calculate Pi Benchmark Using DataframeDACB2468106.436.406.316.27

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - Calculate Pi BenchmarkABDC2040608010080.4279.8179.0978.65

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - Calculate Pi BenchmarkABCD2040608010080.7279.7379.4479.04

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - Calculate Pi BenchmarkABDC2040608010080.8680.0679.8479.27

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Calculate Pi BenchmarkBCDA2040608010080.0979.5678.7978.68

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Calculate Pi BenchmarkBACD2040608010079.7378.8478.7078.50

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - Calculate Pi BenchmarkBADC2040608010080.8880.3979.7279.71

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - Calculate Pi BenchmarkBACD2040608010081.2080.6980.4380.07

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Calculate Pi BenchmarkBDAC2040608010079.5078.8978.8078.44

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - Calculate Pi BenchmarkABDC2040608010080.9680.7980.3979.91

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - Calculate Pi BenchmarkABCD2040608010080.6180.4680.0179.77

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Calculate Pi BenchmarkABDC2040608010080.7480.6780.0679.91

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Calculate Pi Benchmark Using DataframeBADC2468106.036.036.025.97

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Calculate Pi BenchmarkBDCA2040608010079.1178.6478.5478.34

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - Calculate Pi BenchmarkACBD2040608010080.5380.0279.9179.76

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Calculate Pi BenchmarkBDCA2040608010079.7379.2579.1979.16

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - Calculate Pi BenchmarkABCD2040608010079.6879.4479.2779.22

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Calculate Pi BenchmarkABDC2040608010078.8878.8478.6278.56

120 Results Shown

Apache Spark:
  1000000 - 100 - Repartition Test Time
  1000000 - 1000 - Repartition Test Time
  1000000 - 2000 - Broadcast Inner Join Test Time
  1000000 - 100 - Broadcast Inner Join Test Time
  1000000 - 500 - Broadcast Inner Join Test Time
  1000000 - 500 - Repartition Test Time
  1000000 - 100 - Inner Join Test Time
  10000000 - 100 - Repartition Test Time
  1000000 - 1000 - Broadcast Inner Join Test Time
  40000000 - 100 - SHA-512 Benchmark Time
  10000000 - 500 - Group By Test Time
  20000000 - 100 - SHA-512 Benchmark Time
  1000000 - 500 - Inner Join Test Time
  1000000 - 1000 - Inner Join Test Time
  10000000 - 2000 - Broadcast Inner Join Test Time
  1000000 - 2000 - Repartition Test Time
  20000000 - 1000 - Group By Test Time
  20000000 - 500 - Repartition Test Time
SVT-AV1
Apache Spark:
  10000000 - 500 - Repartition Test Time
  10000000 - 1000 - Group By Test Time
  10000000 - 100 - Group By Test Time
SVT-AV1
Apache Spark:
  20000000 - 100 - Repartition Test Time
  1000000 - 1000 - SHA-512 Benchmark Time
  10000000 - 2000 - Inner Join Test Time
  10000000 - 2000 - Group By Test Time
  10000000 - 1000 - Broadcast Inner Join Test Time
  10000000 - 500 - Inner Join Test Time
SVT-AV1
Apache Spark:
  20000000 - 1000 - SHA-512 Benchmark Time
  20000000 - 2000 - Repartition Test Time
  40000000 - 2000 - Inner Join Test Time
SVT-AV1
Apache Spark:
  10000000 - 100 - Inner Join Test Time
  20000000 - 100 - Inner Join Test Time
  20000000 - 100 - Broadcast Inner Join Test Time
  10000000 - 100 - SHA-512 Benchmark Time
  40000000 - 1000 - SHA-512 Benchmark Time
  10000000 - 500 - Broadcast Inner Join Test Time
  20000000 - 100 - Group By Test Time
  1000000 - 2000 - Inner Join Test Time
  1000000 - 500 - SHA-512 Benchmark Time
SVT-AV1
Apache Spark:
  10000000 - 1000 - Repartition Test Time
  40000000 - 100 - Repartition Test Time
  40000000 - 100 - Broadcast Inner Join Test Time
  10000000 - 2000 - SHA-512 Benchmark Time
  1000000 - 2000 - SHA-512 Benchmark Time
  40000000 - 500 - SHA-512 Benchmark Time
  20000000 - 500 - Group By Test Time
  20000000 - 1000 - Repartition Test Time
  20000000 - 500 - Broadcast Inner Join Test Time
  40000000 - 1000 - Group By Test Time
  10000000 - 100 - Broadcast Inner Join Test Time
  40000000 - 100 - Group By Test Time
  10000000 - 1000 - Inner Join Test Time
  20000000 - 500 - SHA-512 Benchmark Time
  40000000 - 2000 - Repartition Test Time
SVT-AV1
Apache Spark:
  1000000 - 1000 - Group By Test Time
  40000000 - 2000 - SHA-512 Benchmark Time
  40000000 - 2000 - Group By Test Time
  20000000 - 500 - Inner Join Test Time
  1000000 - 2000 - Group By Test Time
  1000000 - 1000 - Calculate Pi Benchmark Using Dataframe
  10000000 - 2000 - Repartition Test Time
  20000000 - 2000 - SHA-512 Benchmark Time
  1000000 - 500 - Group By Test Time
  1000000 - 100 - SHA-512 Benchmark Time
  10000000 - 1000 - SHA-512 Benchmark Time
  1000000 - 100 - Group By Test Time
  10000000 - 500 - Calculate Pi Benchmark Using Dataframe
SVT-AV1
Apache Spark:
  1000000 - 500 - Calculate Pi Benchmark Using Dataframe
  40000000 - 2000 - Calculate Pi Benchmark Using Dataframe
  20000000 - 500 - Calculate Pi Benchmark Using Dataframe
  40000000 - 2000 - Broadcast Inner Join Test Time
  1000000 - 2000 - Calculate Pi Benchmark Using Dataframe
  40000000 - 500 - Inner Join Test Time
  20000000 - 2000 - Inner Join Test Time
  10000000 - 1000 - Calculate Pi Benchmark Using Dataframe
  20000000 - 2000 - Group By Test Time
  1000000 - 100 - Calculate Pi Benchmark Using Dataframe
  40000000 - 1000 - Broadcast Inner Join Test Time
  40000000 - 500 - Group By Test Time
  40000000 - 1000 - Calculate Pi Benchmark Using Dataframe
  20000000 - 2000 - Broadcast Inner Join Test Time
  40000000 - 500 - Repartition Test Time
SVT-AV1
Apache Spark:
  10000000 - 2000 - Calculate Pi Benchmark Using Dataframe
  40000000 - 100 - Inner Join Test Time
  40000000 - 1000 - Inner Join Test Time
  20000000 - 1000 - Inner Join Test Time
  20000000 - 1000 - Calculate Pi Benchmark Using Dataframe
  20000000 - 1000 - Broadcast Inner Join Test Time
  40000000 - 500 - Calculate Pi Benchmark Using Dataframe
  40000000 - 100 - Calculate Pi Benchmark Using Dataframe
  40000000 - 500 - Broadcast Inner Join Test Time
  20000000 - 2000 - Calculate Pi Benchmark Using Dataframe
  40000000 - 1000 - Repartition Test Time
  10000000 - 500 - SHA-512 Benchmark Time
  10000000 - 100 - Calculate Pi Benchmark Using Dataframe
  20000000 - 2000 - Calculate Pi Benchmark
  20000000 - 1000 - Calculate Pi Benchmark
  10000000 - 1000 - Calculate Pi Benchmark
  1000000 - 100 - Calculate Pi Benchmark
  1000000 - 2000 - Calculate Pi Benchmark
  40000000 - 100 - Calculate Pi Benchmark
  40000000 - 500 - Calculate Pi Benchmark
  10000000 - 100 - Calculate Pi Benchmark
  40000000 - 1000 - Calculate Pi Benchmark
  40000000 - 2000 - Calculate Pi Benchmark
  10000000 - 2000 - Calculate Pi Benchmark
  20000000 - 100 - Calculate Pi Benchmark Using Dataframe
  1000000 - 500 - Calculate Pi Benchmark
  20000000 - 500 - Calculate Pi Benchmark
  20000000 - 100 - Calculate Pi Benchmark
  10000000 - 500 - Calculate Pi Benchmark
  1000000 - 1000 - Calculate Pi Benchmark