Apache Spark AMD EPYC

2 x AMD EPYC 7713 64-Core testing with a AMD DAYTONA_X (RYM1009B BIOS) and ASPEED on Ubuntu 22.04 via the Phoronix Test Suite.

HTML result view exported from: https://openbenchmarking.org/result/2208047-NE-APACHESPA20&grs&sor.

Apache Spark AMD EPYCProcessorMotherboardChipsetMemoryDiskGraphicsMonitorNetworkOSKernelDesktopDisplay ServerVulkanCompilerFile-SystemScreen ResolutionABC2 x AMD EPYC 7713 64-Core @ 2.00GHz (128 Cores / 256 Threads)AMD DAYTONA_X (RYM1009B BIOS)AMD Starship/Matisse512GB3841GB Micron_9300_MTFDHAL3T8TDPASPEEDVE2282 x Mellanox MT27710Ubuntu 22.045.19.0-051900daily20220803-generic (x86_64)GNOME Shell 42.2X Server 1.21.1.31.2.204GCC 11.2.0ext41920x1080OpenBenchmarking.orgKernel Details- Transparent Huge Pages: madviseProcessor Details- Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0xa001173 Java Details- OpenJDK Runtime Environment (build 11.0.15+10-Ubuntu-0ubuntu0.22.04.1)Python Details- Python 3.10.4Security Details- itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling + srbds: Not affected + tsx_async_abort: Not affected

Apache Spark AMD EPYCspark: 10000000 - 2000 - Calculate Pi Benchmark Using Dataframespark: 1000000 - 500 - Calculate Pi Benchmark Using Dataframespark: 20000000 - 100 - Inner Join Test Timespark: 20000000 - 100 - Broadcast Inner Join Test Timespark: 1000000 - 500 - SHA-512 Benchmark Timespark: 1000000 - 100 - Inner Join Test Timespark: 10000000 - 2000 - SHA-512 Benchmark Timespark: 1000000 - 1000 - SHA-512 Benchmark Timespark: 20000000 - 100 - SHA-512 Benchmark Timespark: 10000000 - 100 - SHA-512 Benchmark Timespark: 20000000 - 2000 - SHA-512 Benchmark Timespark: 1000000 - 100 - Group By Test Timespark: 10000000 - 1000 - SHA-512 Benchmark Timespark: 10000000 - 500 - SHA-512 Benchmark Timespark: 20000000 - 100 - Group By Test Timespark: 1000000 - 2000 - SHA-512 Benchmark Timespark: 10000000 - 2000 - Calculate Pi Benchmarkspark: 1000000 - 100 - SHA-512 Benchmark Timespark: 20000000 - 1000 - SHA-512 Benchmark Timespark: 40000000 - 100 - SHA-512 Benchmark Timespark: 40000000 - 1000 - Calculate Pi Benchmarkspark: 40000000 - 1000 - SHA-512 Benchmark Timespark: 1000000 - 500 - Calculate Pi Benchmarkspark: 20000000 - 100 - Repartition Test Timespark: 40000000 - 500 - Calculate Pi Benchmarkspark: 20000000 - 500 - Calculate Pi Benchmarkspark: 40000000 - 2000 - SHA-512 Benchmark Timespark: 1000000 - 1000 - Calculate Pi Benchmarkspark: 40000000 - 2000 - Calculate Pi Benchmarkspark: 20000000 - 2000 - Calculate Pi Benchmarkspark: 1000000 - 2000 - Calculate Pi Benchmarkspark: 1000000 - 100 - Calculate Pi Benchmarkspark: 20000000 - 100 - Calculate Pi Benchmarkspark: 40000000 - 100 - Calculate Pi Benchmarkspark: 20000000 - 500 - SHA-512 Benchmark Timespark: 10000000 - 1000 - Calculate Pi Benchmarkspark: 40000000 - 500 - SHA-512 Benchmark Timespark: 10000000 - 100 - Calculate Pi Benchmarkspark: 20000000 - 1000 - Calculate Pi Benchmarkspark: 10000000 - 500 - Calculate Pi Benchmarkspark: 10000000 - 2000 - Broadcast Inner Join Test Timespark: 10000000 - 2000 - Inner Join Test Timespark: 10000000 - 2000 - Repartition Test Timespark: 10000000 - 2000 - Group By Test Timespark: 1000000 - 500 - Broadcast Inner Join Test Timespark: 1000000 - 500 - Inner Join Test Timespark: 1000000 - 500 - Repartition Test Timespark: 1000000 - 500 - Group By Test Timespark: 40000000 - 2000 - Calculate Pi Benchmark Using Dataframespark: 40000000 - 1000 - Calculate Pi Benchmark Using Dataframespark: 20000000 - 2000 - Calculate Pi Benchmark Using Dataframespark: 20000000 - 1000 - Calculate Pi Benchmark Using Dataframespark: 10000000 - 1000 - Calculate Pi Benchmark Using Dataframespark: 40000000 - 500 - Calculate Pi Benchmark Using Dataframespark: 40000000 - 100 - Calculate Pi Benchmark Using Dataframespark: 20000000 - 500 - Calculate Pi Benchmark Using Dataframespark: 20000000 - 100 - Calculate Pi Benchmark Using Dataframespark: 10000000 - 500 - Calculate Pi Benchmark Using Dataframespark: 10000000 - 100 - Broadcast Inner Join Test Timespark: 10000000 - 100 - Inner Join Test Timespark: 10000000 - 100 - Repartition Test Timespark: 10000000 - 100 - Group By Test Timespark: 10000000 - 100 - Calculate Pi Benchmark Using Dataframespark: 1000000 - 2000 - Broadcast Inner Join Test Timespark: 1000000 - 2000 - Inner Join Test Timespark: 1000000 - 2000 - Repartition Test Timespark: 1000000 - 2000 - Group By Test Timespark: 1000000 - 2000 - Calculate Pi Benchmark Using Dataframespark: 1000000 - 1000 - Broadcast Inner Join Test Timespark: 1000000 - 1000 - Inner Join Test Timespark: 1000000 - 1000 - Repartition Test Timespark: 1000000 - 1000 - Group By Test Timespark: 1000000 - 1000 - Calculate Pi Benchmark Using Dataframespark: 1000000 - 100 - Broadcast Inner Join Test Timespark: 1000000 - 100 - Repartition Test Timespark: 1000000 - 100 - Calculate Pi Benchmark Using DataframeABC3.343.185.932.9213.195.9118.8311.6020.966.5312.9113.295.9117.484.5420.9534.9617.3335.1817.4717.6717.4535.7817.4517.6317.4217.5217.5817.3817.5420.3717.4435.2617.4817.4717.452.042.903.673.343.543.383.443.444.253.805.818.517.2812.323.514.427.186.0411.053.213.906.585.4911.953.381.862.943.673.223.2214.2512.026.013.1713.096.3619.2011.6320.476.7813.0912.8537.616.1617.424.6220.6535.1217.3135.5517.4913.4817.4117.6435.3617.5917.5417.3017.5517.5317.4017.5120.4617.5435.0317.3717.5017.498.1812.7614.4418.194.495.594.8715.623.433.553.623.573.223.283.093.823.393.316.239.128.3713.263.684.047.345.9810.963.643.976.936.2514.252.461.842.753.124.092.6912.7610.876.063.2112.626.0819.3312.2421.746.9112.5312.72350245839.265.9117.804.4521.4233.9417.8436.2117.1713.8617.2817.2635.0717.8017.3117.6117.6917.7517.1917.7120.2617.6135.2817.37897353517.4317.484.367029263.803.332.562.893.972.533.403.403.086.039.278.1612.672.044.447.478.5111.863.254.137.016.1611.053.031.992.913.48OpenBenchmarking.org

Apache Spark

Row Count: 10000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Calculate Pi Benchmark Using DataframeCBA0.92031.84062.76093.68124.6015SE +/- 0.18, N = 15SE +/- 0.22, N = 152.563.223.34

Apache Spark

Row Count: 1000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Calculate Pi Benchmark Using DataframeCAB0.85281.70562.55843.41124.264SE +/- 0.16, N = 15SE +/- 0.16, N = 152.693.183.22

Apache Spark

Row Count: 20000000 - Partitions: 100 - Inner Join Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Inner Join Test TimeCB48121620SE +/- 0.76, N = 212.7614.25

Apache Spark

Row Count: 20000000 - Partitions: 100 - Broadcast Inner Join Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Broadcast Inner Join Test TimeCB3691215SE +/- 0.81, N = 210.8712.02

Apache Spark

Row Count: 1000000 - Partitions: 500 - SHA-512 Benchmark Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - SHA-512 Benchmark TimeCAB246810SE +/- 0.09, N = 15SE +/- 0.09, N = 155.545.936.01

Apache Spark

Row Count: 1000000 - Partitions: 100 - Inner Join Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Inner Join Test TimeABC0.72231.44462.16692.88923.6115SE +/- 0.04, N = 3SE +/- 0.03, N = 42.923.173.21

Apache Spark

Row Count: 10000000 - Partitions: 2000 - SHA-512 Benchmark Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - SHA-512 Benchmark TimeCBA48121620SE +/- 0.12, N = 15SE +/- 0.15, N = 1512.6213.0913.19

Apache Spark

Row Count: 1000000 - Partitions: 1000 - SHA-512 Benchmark Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - SHA-512 Benchmark TimeACB246810SE +/- 0.06, N = 15SE +/- 0.06, N = 35.916.086.36

Apache Spark

Row Count: 20000000 - Partitions: 100 - SHA-512 Benchmark Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - SHA-512 Benchmark TimeABC510152025SE +/- 0.20, N = 3SE +/- 0.17, N = 1218.8319.2019.33

Apache Spark

Row Count: 10000000 - Partitions: 100 - SHA-512 Benchmark Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - SHA-512 Benchmark TimeCAB3691215SE +/- 0.09, N = 3SE +/- 0.11, N = 1511.5011.6011.63

Apache Spark

Row Count: 20000000 - Partitions: 2000 - SHA-512 Benchmark Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - SHA-512 Benchmark TimeBAC510152025SE +/- 0.25, N = 3SE +/- 0.18, N = 1220.4720.9621.74

Apache Spark

Row Count: 1000000 - Partitions: 100 - Group By Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Group By Test TimeABC246810SE +/- 0.04, N = 3SE +/- 0.02, N = 46.536.786.91

Apache Spark

Row Count: 10000000 - Partitions: 1000 - SHA-512 Benchmark Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - SHA-512 Benchmark TimeCAB3691215SE +/- 0.10, N = 12SE +/- 0.15, N = 1212.5312.9113.09

Apache Spark

Row Count: 10000000 - Partitions: 500 - SHA-512 Benchmark Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - SHA-512 Benchmark TimeCBA3691215SE +/- 0.13, N = 5SE +/- 0.15, N = 312.7212.8513.29

Apache Spark

Row Count: 20000000 - Partitions: 100 - Group By Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Group By Test TimeBC918273645SE +/- 3.13, N = 237.6139.26

Apache Spark

Row Count: 1000000 - Partitions: 2000 - SHA-512 Benchmark Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - SHA-512 Benchmark TimeACB246810SE +/- 0.07, N = 15SE +/- 0.05, N = 85.915.916.16

Apache Spark

Row Count: 10000000 - Partitions: 2000 - Calculate Pi Benchmark

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Calculate Pi BenchmarkCBA48121620SE +/- 0.07, N = 15SE +/- 0.05, N = 1517.1117.4217.48

Apache Spark

Row Count: 1000000 - Partitions: 100 - SHA-512 Benchmark Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - SHA-512 Benchmark TimeCAB1.03952.0793.11854.1585.1975SE +/- 0.03, N = 3SE +/- 0.06, N = 44.454.544.62

Apache Spark

Row Count: 20000000 - Partitions: 1000 - SHA-512 Benchmark Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - SHA-512 Benchmark TimeBAC510152025SE +/- 0.28, N = 3SE +/- 0.23, N = 520.6520.9521.42

Apache Spark

Row Count: 40000000 - Partitions: 100 - SHA-512 Benchmark Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - SHA-512 Benchmark TimeCAB816243240SE +/- 0.27, N = 12SE +/- 0.38, N = 933.9434.9635.12

Apache Spark

Row Count: 40000000 - Partitions: 1000 - Calculate Pi Benchmark

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - Calculate Pi BenchmarkBAC48121620SE +/- 0.01, N = 3SE +/- 0.07, N = 517.3117.3317.84

Apache Spark

Row Count: 40000000 - Partitions: 1000 - SHA-512 Benchmark Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - SHA-512 Benchmark TimeABC816243240SE +/- 0.38, N = 5SE +/- 0.50, N = 335.1835.5536.21

Apache Spark

Row Count: 1000000 - Partitions: 500 - Calculate Pi Benchmark

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Calculate Pi BenchmarkCAB48121620SE +/- 0.07, N = 15SE +/- 0.07, N = 1517.1717.4717.49

Apache Spark

Row Count: 20000000 - Partitions: 100 - Repartition Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Repartition Test TimeBC48121620SE +/- 0.08, N = 213.4813.86

Apache Spark

Row Count: 40000000 - Partitions: 500 - Calculate Pi Benchmark

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - Calculate Pi BenchmarkCBA48121620SE +/- 0.07, N = 5SE +/- 0.09, N = 317.2817.4117.67

Apache Spark

Row Count: 20000000 - Partitions: 500 - Calculate Pi Benchmark

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - Calculate Pi BenchmarkCAB48121620SE +/- 0.18, N = 3SE +/- 0.10, N = 317.2617.4517.64

Apache Spark

Row Count: 40000000 - Partitions: 2000 - SHA-512 Benchmark Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - SHA-512 Benchmark TimeCBA816243240SE +/- 0.39, N = 5SE +/- 0.09, N = 335.0735.3635.78

Apache Spark

Row Count: 1000000 - Partitions: 1000 - Calculate Pi Benchmark

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Calculate Pi BenchmarkABC48121620SE +/- 0.06, N = 15SE +/- 0.19, N = 317.4517.5917.80

Apache Spark

Row Count: 40000000 - Partitions: 2000 - Calculate Pi Benchmark

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - Calculate Pi BenchmarkCBA48121620SE +/- 0.10, N = 5SE +/- 0.16, N = 317.3117.5417.63

Apache Spark

Row Count: 20000000 - Partitions: 2000 - Calculate Pi Benchmark

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - Calculate Pi BenchmarkBAC48121620SE +/- 0.06, N = 3SE +/- 0.06, N = 1217.3017.4217.61

Apache Spark

Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Calculate Pi BenchmarkCAB48121620SE +/- 0.07, N = 15SE +/- 0.09, N = 817.4617.5217.55

Apache Spark

Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Calculate Pi BenchmarkBAC48121620SE +/- 0.13, N = 4SE +/- 0.20, N = 317.5317.5817.75

Apache Spark

Row Count: 20000000 - Partitions: 100 - Calculate Pi Benchmark

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Calculate Pi BenchmarkCAB48121620SE +/- 0.16, N = 3SE +/- 0.06, N = 1217.1917.3817.40

Apache Spark

Row Count: 40000000 - Partitions: 100 - Calculate Pi Benchmark

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - Calculate Pi BenchmarkBAC48121620SE +/- 0.10, N = 9SE +/- 0.06, N = 1217.5117.5417.71

Apache Spark

Row Count: 20000000 - Partitions: 500 - SHA-512 Benchmark Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - SHA-512 Benchmark TimeCAB510152025SE +/- 0.21, N = 3SE +/- 0.13, N = 320.2620.3720.46

Apache Spark

Row Count: 10000000 - Partitions: 1000 - Calculate Pi Benchmark

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - Calculate Pi BenchmarkABC48121620SE +/- 0.07, N = 12SE +/- 0.13, N = 1217.4417.5417.61

Apache Spark

Row Count: 40000000 - Partitions: 500 - SHA-512 Benchmark Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - SHA-512 Benchmark TimeBAC816243240SE +/- 0.38, N = 5SE +/- 0.33, N = 335.0335.2635.28

Apache Spark

Row Count: 10000000 - Partitions: 100 - Calculate Pi Benchmark

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Calculate Pi BenchmarkBCA48121620SE +/- 0.05, N = 15SE +/- 0.11, N = 317.3717.3817.48

Apache Spark

Row Count: 20000000 - Partitions: 1000 - Calculate Pi Benchmark

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - Calculate Pi BenchmarkCAB48121620SE +/- 0.11, N = 5SE +/- 0.12, N = 317.4317.4717.50

Apache Spark

Row Count: 10000000 - Partitions: 500 - Calculate Pi Benchmark

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - Calculate Pi BenchmarkACB48121620SE +/- 0.11, N = 3SE +/- 0.13, N = 517.4517.4817.49

Apache Spark

Row Count: 10000000 - Partitions: 2000 - Broadcast Inner Join Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Broadcast Inner Join Test TimeB246810SE +/- 0.39, N = 28.18

Apache Spark

Row Count: 10000000 - Partitions: 2000 - Inner Join Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Inner Join Test TimeB3691215SE +/- 1.47, N = 212.76

Apache Spark

Row Count: 10000000 - Partitions: 2000 - Repartition Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Repartition Test TimeB48121620SE +/- 0.97, N = 214.44

Apache Spark

Row Count: 10000000 - Partitions: 2000 - Group By Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 2000 - Group By Test TimeB48121620SE +/- 0.52, N = 218.19

Apache Spark

Row Count: 1000000 - Partitions: 500 - Broadcast Inner Join Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Broadcast Inner Join Test TimeB1.01032.02063.03094.04125.0515SE +/- 0.46, N = 24.49

Apache Spark

Row Count: 1000000 - Partitions: 500 - Inner Join Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Inner Join Test TimeB1.25782.51563.77345.03126.289SE +/- 0.17, N = 25.59

Apache Spark

Row Count: 1000000 - Partitions: 500 - Repartition Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Repartition Test TimeB1.09582.19163.28744.38325.479SE +/- 0.07, N = 24.87

Apache Spark

Row Count: 1000000 - Partitions: 500 - Group By Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 500 - Group By Test TimeB48121620SE +/- 4.11, N = 215.62

Apache Spark

Row Count: 40000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 2000 - Calculate Pi Benchmark Using DataframeABC0.98261.96522.94783.93044.913SE +/- 0.01771345, N = 3SE +/- 0.23677806, N = 52.040000003.430000004.36702926

Apache Spark

Row Count: 40000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 1000 - Calculate Pi Benchmark Using DataframeABC0.8551.712.5653.424.275SE +/- 0.35, N = 5SE +/- 0.36, N = 32.903.553.80

Apache Spark

Row Count: 20000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 2000 - Calculate Pi Benchmark Using DataframeCBA0.82581.65162.47743.30324.129SE +/- 0.57, N = 3SE +/- 0.23, N = 123.333.623.67

Apache Spark

Row Count: 20000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 1000 - Calculate Pi Benchmark Using DataframeCAB0.80331.60662.40993.21324.0165SE +/- 0.33, N = 5SE +/- 0.29, N = 32.563.343.57

Apache Spark

Row Count: 10000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 1000 - Calculate Pi Benchmark Using DataframeCBA0.79651.5932.38953.1863.9825SE +/- 0.15, N = 12SE +/- 0.19, N = 122.893.223.54

Apache Spark

Row Count: 40000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 500 - Calculate Pi Benchmark Using DataframeBAC0.89331.78662.67993.57324.4665SE +/- 0.36, N = 5SE +/- 0.34, N = 33.283.383.97

Apache Spark

Row Count: 40000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 40000000 - Partitions: 100 - Calculate Pi Benchmark Using DataframeCBA0.7741.5482.3223.0963.87SE +/- 0.24, N = 9SE +/- 0.20, N = 122.533.093.44

Apache Spark

Row Count: 20000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 500 - Calculate Pi Benchmark Using DataframeCAB0.85951.7192.57853.4384.2975SE +/- 0.33, N = 3SE +/- 0.64, N = 33.403.443.82

Apache Spark

Row Count: 20000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 20000000 - Partitions: 100 - Calculate Pi Benchmark Using DataframeCBA0.95631.91262.86893.82524.7815SE +/- 0.21, N = 12SE +/- 0.03, N = 32.903.394.25

Apache Spark

Row Count: 10000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 500 - Calculate Pi Benchmark Using DataframeCBA0.8551.712.5653.424.275SE +/- 0.46, N = 5SE +/- 0.30, N = 33.083.313.80

Apache Spark

Row Count: 10000000 - Partitions: 100 - Broadcast Inner Join Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Broadcast Inner Join Test TimeACB246810SE +/- 0.24, N = 3SE +/- 0.13, N = 115.816.036.23

Apache Spark

Row Count: 10000000 - Partitions: 100 - Inner Join Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Inner Join Test TimeABC3691215SE +/- 0.09, N = 3SE +/- 0.18, N = 118.519.129.27

Apache Spark

Row Count: 10000000 - Partitions: 100 - Repartition Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Repartition Test TimeACB246810SE +/- 0.41, N = 3SE +/- 0.16, N = 117.288.168.37

Apache Spark

Row Count: 10000000 - Partitions: 100 - Group By Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Group By Test TimeACB3691215SE +/- 0.23, N = 3SE +/- 0.27, N = 1112.3212.6713.26

Apache Spark

Row Count: 10000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 10000000 - Partitions: 100 - Calculate Pi Benchmark Using DataframeCAB0.8281.6562.4843.3124.14SE +/- 0.13, N = 3SE +/- 0.13, N = 152.043.513.68

Apache Spark

Row Count: 1000000 - Partitions: 2000 - Broadcast Inner Join Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Broadcast Inner Join Test TimeBAC0.9991.9982.9973.9964.995SE +/- 0.12, N = 6SE +/- 0.11, N = 114.044.424.44

Apache Spark

Row Count: 1000000 - Partitions: 2000 - Inner Join Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Inner Join Test TimeABC246810SE +/- 0.22, N = 11SE +/- 0.23, N = 67.187.347.47

Apache Spark

Row Count: 1000000 - Partitions: 2000 - Repartition Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Repartition Test TimeBAC246810SE +/- 0.25, N = 6SE +/- 0.22, N = 115.986.048.51

Apache Spark

Row Count: 1000000 - Partitions: 2000 - Group By Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Group By Test TimeBAC3691215SE +/- 0.32, N = 6SE +/- 0.24, N = 1110.9611.0511.86

Apache Spark

Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark Using DataframeACB0.8191.6382.4573.2764.095SE +/- 0.18, N = 15SE +/- 0.27, N = 83.213.253.64

Apache Spark

Row Count: 1000000 - Partitions: 1000 - Broadcast Inner Join Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Broadcast Inner Join Test TimeABC0.92931.85862.78793.71724.6465SE +/- 0.10, N = 9SE +/- 0.30, N = 23.903.974.13

Apache Spark

Row Count: 1000000 - Partitions: 1000 - Inner Join Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Inner Join Test TimeABC246810SE +/- 0.25, N = 9SE +/- 0.77, N = 26.586.937.01

Apache Spark

Row Count: 1000000 - Partitions: 1000 - Repartition Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Repartition Test TimeACB246810SE +/- 0.16, N = 9SE +/- 0.43, N = 25.496.166.25

Apache Spark

Row Count: 1000000 - Partitions: 1000 - Group By Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Group By Test TimeCAB48121620SE +/- 0.70, N = 9SE +/- 2.86, N = 211.0511.9514.25

Apache Spark

Row Count: 1000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 1000 - Calculate Pi Benchmark Using DataframeBCA0.76051.5212.28153.0423.8025SE +/- 0.30, N = 3SE +/- 0.21, N = 152.463.033.38

Apache Spark

Row Count: 1000000 - Partitions: 100 - Broadcast Inner Join Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Broadcast Inner Join Test TimeBAC0.44780.89561.34341.79122.239SE +/- 0.05, N = 4SE +/- 0.08, N = 31.841.861.99

Apache Spark

Row Count: 1000000 - Partitions: 100 - Repartition Test Time

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Repartition Test TimeBCA0.66151.3231.98452.6463.3075SE +/- 0.07, N = 4SE +/- 0.12, N = 32.752.912.94

Apache Spark

Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe

OpenBenchmarking.orgSeconds, Fewer Is BetterApache Spark 3.3Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark Using DataframeBCA0.82581.65162.47743.30324.129SE +/- 0.31, N = 4SE +/- 0.32, N = 33.123.483.67

Phoronix Test Suite v10.8.5