apache spark 5800X3D AMD Ryzen 7 5800X3D 8-Core testing with a ASUS ROG CROSSHAIR VIII HERO (4201 BIOS) and Gigabyte AMD Radeon RX 6600 8GB on Ubuntu 22.04 via the Phoronix Test Suite.
HTML result view exported from: https://openbenchmarking.org/result/2208042-PTS-APACHESP16&sor&grs .
apache spark 5800X3D Processor Motherboard Chipset Memory Disk Graphics Audio Monitor Network OS Kernel Desktop Display Server OpenGL Vulkan Compiler File-System Screen Resolution A B C AMD Ryzen 7 5800X3D 8-Core @ 3.40GHz (8 Cores / 16 Threads) ASUS ROG CROSSHAIR VIII HERO (4201 BIOS) AMD Starship/Matisse 32GB 1000GB Western Digital WDS100T1X0E-00AFY0 Gigabyte AMD Radeon RX 6600 8GB (2750/875MHz) AMD Navi 21 HDMI Audio ASUS MG28U Realtek RTL8125 2.5GbE + Intel I211 Ubuntu 22.04 5.19.0-rc6-drm-next-28july (x86_64) GNOME Shell 42.2 X Server 1.21.1.3 + Wayland 4.6 Mesa 22.2.0-devel (git-c6ded153d1) (LLVM 12.0.0 DRM 3.48) 1.3.211 GCC 11.2.0 ext4 3840x2160 OpenBenchmarking.org Kernel Details - Transparent Huge Pages: madvise Processor Details - Scaling Governor: acpi-cpufreq schedutil (Boost: Enabled) - CPU Microcode: 0xa20120a Java Details - OpenJDK Runtime Environment (build 11.0.15+10-Ubuntu-0ubuntu0.22.04.1) Python Details - Python 3.10.4 Security Details - itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling + srbds: Not affected + tsx_async_abort: Not affected
apache spark 5800X3D spark: 1000000 - 1000 - Inner Join Test Time spark: 1000000 - 2000 - Inner Join Test Time spark: 20000000 - 500 - Inner Join Test Time spark: 20000000 - 1000 - Broadcast Inner Join Test Time spark: 1000000 - 2000 - Broadcast Inner Join Test Time spark: 40000000 - 100 - Inner Join Test Time spark: 10000000 - 500 - Broadcast Inner Join Test Time spark: 10000000 - 2000 - Inner Join Test Time spark: 40000000 - 500 - Group By Test Time spark: 1000000 - 500 - Group By Test Time spark: 10000000 - 100 - Inner Join Test Time spark: 20000000 - 2000 - Broadcast Inner Join Test Time spark: 10000000 - 500 - Inner Join Test Time spark: 20000000 - 500 - Repartition Test Time spark: 1000000 - 2000 - Repartition Test Time spark: 1000000 - 100 - Repartition Test Time spark: 40000000 - 100 - Broadcast Inner Join Test Time spark: 1000000 - 2000 - Group By Test Time spark: 40000000 - 500 - SHA-512 Benchmark Time spark: 20000000 - 1000 - Inner Join Test Time spark: 20000000 - 100 - Broadcast Inner Join Test Time spark: 40000000 - 2000 - Inner Join Test Time spark: 10000000 - 100 - Broadcast Inner Join Test Time spark: 1000000 - 100 - SHA-512 Benchmark Time spark: 40000000 - 2000 - SHA-512 Benchmark Time spark: 40000000 - 500 - Calculate Pi Benchmark spark: 10000000 - 500 - Group By Test Time spark: 40000000 - 100 - Repartition Test Time spark: 10000000 - 100 - Repartition Test Time spark: 40000000 - 1000 - Inner Join Test Time spark: 10000000 - 500 - Repartition Test Time spark: 40000000 - 1000 - Calculate Pi Benchmark spark: 20000000 - 1000 - Group By Test Time spark: 20000000 - 2000 - Repartition Test Time spark: 1000000 - 1000 - Group By Test Time spark: 40000000 - 2000 - Broadcast Inner Join Test Time spark: 20000000 - 500 - Broadcast Inner Join Test Time spark: 1000000 - 100 - Calculate Pi Benchmark spark: 1000000 - 1000 - Repartition Test Time spark: 10000000 - 2000 - Broadcast Inner Join Test Time spark: 10000000 - 1000 - Group By Test Time spark: 40000000 - 2000 - Repartition Test Time spark: 1000000 - 100 - Calculate Pi Benchmark Using Dataframe spark: 10000000 - 1000 - Repartition Test Time spark: 40000000 - 500 - Repartition Test Time spark: 1000000 - 500 - Calculate Pi Benchmark Using Dataframe spark: 1000000 - 1000 - SHA-512 Benchmark Time spark: 40000000 - 500 - Inner Join Test Time spark: 10000000 - 2000 - Group By Test Time spark: 20000000 - 2000 - Group By Test Time spark: 40000000 - 1000 - Repartition Test Time spark: 40000000 - 1000 - Group By Test Time spark: 1000000 - 500 - Repartition Test Time spark: 20000000 - 100 - Group By Test Time spark: 20000000 - 2000 - Inner Join Test Time spark: 10000000 - 1000 - Calculate Pi Benchmark Using Dataframe spark: 20000000 - 500 - SHA-512 Benchmark Time spark: 20000000 - 100 - Calculate Pi Benchmark Using Dataframe spark: 40000000 - 2000 - Group By Test Time spark: 10000000 - 2000 - Calculate Pi Benchmark Using Dataframe spark: 10000000 - 100 - Group By Test Time spark: 20000000 - 100 - Repartition Test Time spark: 1000000 - 500 - SHA-512 Benchmark Time spark: 10000000 - 2000 - Repartition Test Time spark: 10000000 - 1000 - SHA-512 Benchmark Time spark: 20000000 - 100 - Inner Join Test Time spark: 40000000 - 500 - Broadcast Inner Join Test Time spark: 10000000 - 2000 - Calculate Pi Benchmark spark: 1000000 - 500 - Calculate Pi Benchmark spark: 40000000 - 2000 - Calculate Pi Benchmark Using Dataframe spark: 1000000 - 1000 - Calculate Pi Benchmark Using Dataframe spark: 20000000 - 2000 - Calculate Pi Benchmark spark: 1000000 - 2000 - Calculate Pi Benchmark spark: 20000000 - 1000 - Calculate Pi Benchmark Using Dataframe spark: 10000000 - 500 - SHA-512 Benchmark Time spark: 20000000 - 1000 - Calculate Pi Benchmark spark: 10000000 - 100 - SHA-512 Benchmark Time spark: 1000000 - 2000 - Calculate Pi Benchmark Using Dataframe spark: 20000000 - 500 - Group By Test Time spark: 40000000 - 1000 - Calculate Pi Benchmark Using Dataframe spark: 40000000 - 100 - Calculate Pi Benchmark spark: 40000000 - 1000 - SHA-512 Benchmark Time spark: 10000000 - 500 - Calculate Pi Benchmark spark: 20000000 - 500 - Calculate Pi Benchmark spark: 1000000 - 500 - Inner Join Test Time spark: 1000000 - 1000 - Calculate Pi Benchmark spark: 20000000 - 2000 - Calculate Pi Benchmark Using Dataframe spark: 1000000 - 2000 - SHA-512 Benchmark Time spark: 40000000 - 100 - SHA-512 Benchmark Time spark: 40000000 - 1000 - Broadcast Inner Join Test Time spark: 20000000 - 1000 - Repartition Test Time spark: 10000000 - 100 - Calculate Pi Benchmark spark: 40000000 - 2000 - Calculate Pi Benchmark spark: 20000000 - 100 - Calculate Pi Benchmark spark: 40000000 - 100 - Group By Test Time spark: 20000000 - 500 - Calculate Pi Benchmark Using Dataframe spark: 40000000 - 500 - Calculate Pi Benchmark Using Dataframe spark: 1000000 - 100 - Group By Test Time spark: 10000000 - 1000 - Calculate Pi Benchmark spark: 20000000 - 1000 - SHA-512 Benchmark Time spark: 20000000 - 100 - SHA-512 Benchmark Time spark: 20000000 - 2000 - SHA-512 Benchmark Time spark: 40000000 - 100 - Calculate Pi Benchmark Using Dataframe spark: 10000000 - 100 - Calculate Pi Benchmark Using Dataframe spark: 10000000 - 500 - Calculate Pi Benchmark Using Dataframe spark: 10000000 - 2000 - SHA-512 Benchmark Time spark: 10000000 - 1000 - Broadcast Inner Join Test Time spark: 10000000 - 1000 - Inner Join Test Time spark: 1000000 - 1000 - Broadcast Inner Join Test Time spark: 1000000 - 500 - Broadcast Inner Join Test Time spark: 1000000 - 100 - Broadcast Inner Join Test Time spark: 1000000 - 100 - Inner Join Test Time A B C 1.92 2.32 20.035527670 19.87 2.02 40.00 10.589748727 11.20 23.97 3.15 11.340071822 20.01 10.320102526 16.94 2.50 1.90 40.78 3.74 50.13 19.97 22.08 40.60 11.566249356 2.91 50.33 151.330983793 6.75 33.933058393 9.16 39.60 8.94 149.450658423 10.25 17.52 3.42 41.55 20.35 149.348708414 2.20 10.60 6.89 34.56 8.67 9.05 33.546966500 8.69 3.17 39.55 7.03 10.25 34.50 22.89 1.99 10.65 20.79 8.49 25.771477402 8.67 23.26 8.58 6.60 17.908074200 3.06 9.34 13.97 21.82 40.32 148.548505705 150.184683924 8.59 8.68 148.853580559 150.031645092 8.62 13.94 149.230168484 14.539789898 8.68 10.118608339 8.61 149.227508787 49.63 148.982085231 148.224458268 1.63 150.409490333 8.64 3.46 49.41 40.993090142 17.03 149.758709975 148.601893741 149.362186051 27.362051435 8.60 8.59 2.89 148.548472344 25.55 27.22 25.70 8.63 8.66 8.69 14.15 10.65 10.79 1.58 1.48 1.31 1.45 1.84 2.36 19.77 20.66 2.05 39.87 10.11 11.21 23.49 3.20 11.01 19.87 10.48 17.02 2.46 1.93 40.14 3.84 49.86 20.33 22.27 39.58 11.28 2.88 51.35 149.128695060 6.82 34.41 9.15 40.50 9.03 148.664434137 10.16 17.81 3.43 40.71 20.05 148.735383131 2.20 10.56 6.77 33.98 8.61 9.01 33.85 8.67 3.19 40.14 7.14 10.32 34.09 23.24 1.99 10.50 20.72 8.54 25.98 8.59 22.94 8.57 6.64 17.81 3.02 9.22 13.89 22.08 40.760680072 149.183387153 149.134613280 8.57 8.61 148.300590012 148.583093415 8.58 13.97 147.914605982 14.60 8.61 10.05 8.62 148.597175671 49.95 148.915757535 149.139980735 1.63 149.520497364 8.60 3.47 49.28 41.01 17.09 149.564826029 148.502716195 148.786767111 27.28 8.60 8.61 2.88 148.532731450 25.53 27.24 25.76 8.63 8.66 8.68 14.12 10.60 11.02 1.66 1.45 1.36 1.40 1.76 2.50 21.21 21.08 1.94 41.98 10.24 11.66 23.04 3.27 10.93 19.31 10.14 17.493210745 2.54 1.96 39.621362405 3.79 48.84 20.49 22.650420462 39.848198836 11.54 2.84 50.12 147.715180363 6.66 34.74 8.95 39.899100315 8.83 151.941134869 10.03 17.44 3.36 41.18 19.94 146.52097559 2.16 10.411906333 6.86 33.96 8.52 8.90 34.11 8.55 3.14 40.17 7.04 10.41 34.62 23.18 2.02 10.60 20.50 8.61 25.62 8.71 23.170788559 8.687892725 6.55 18.05 3.03 9.24 14.07 22.09 40.291389364 150.275810804 148.565766046 8.50 8.59 149.847002868 149.71767697 8.54 14.07 149.292751295 14.47 8.63 10.04 8.56 149.610888179 49.84 148.055891825 148.939367323 1.62 149.529773024 8.59 3.48 49.53 41.20 17.01 149.06749502 147.981380284 149.245823899 27.382169172 8.57 8.58 2.88 148.988566083 25.60 27.29 25.72 8.65 8.64 8.67 14.14 10.08 10.51 1.74 1.38 1.30 1.44 OpenBenchmarking.org
Apache Spark Row Count: 1000000 - Partitions: 1000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Inner Join Test Time C B A 0.432 0.864 1.296 1.728 2.16 SE +/- 0.02, N = 12 SE +/- 0.01, N = 3 1.76 1.84 1.92
Apache Spark Row Count: 1000000 - Partitions: 2000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Inner Join Test Time A B C 0.5625 1.125 1.6875 2.25 2.8125 SE +/- 0.07, N = 3 SE +/- 0.07, N = 3 2.32 2.36 2.50
Apache Spark Row Count: 20000000 - Partitions: 500 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Inner Join Test Time B A C 5 10 15 20 25 SE +/- 0.18, N = 4 SE +/- 0.21, N = 3 19.77 20.04 21.21
Apache Spark Row Count: 20000000 - Partitions: 1000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Broadcast Inner Join Test Time A B C 5 10 15 20 25 SE +/- 0.23, N = 3 SE +/- 0.37, N = 3 19.87 20.66 21.08
Apache Spark Row Count: 1000000 - Partitions: 2000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Broadcast Inner Join Test Time C A B 0.4613 0.9226 1.3839 1.8452 2.3065 SE +/- 0.04, N = 3 SE +/- 0.05, N = 3 1.94 2.02 2.05
Apache Spark Row Count: 40000000 - Partitions: 100 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Inner Join Test Time B A C 10 20 30 40 50 SE +/- 0.42, N = 3 SE +/- 0.25, N = 3 39.87 40.00 41.98
Apache Spark Row Count: 10000000 - Partitions: 500 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Broadcast Inner Join Test Time B C A 3 6 9 12 15 SE +/- 0.04, N = 3 SE +/- 0.16, N = 3 10.11 10.24 10.59
Apache Spark Row Count: 10000000 - Partitions: 2000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Inner Join Test Time A B C 3 6 9 12 15 SE +/- 0.18, N = 3 SE +/- 0.16, N = 3 11.20 11.21 11.66
Apache Spark Row Count: 40000000 - Partitions: 500 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Group By Test Time C B A 6 12 18 24 30 SE +/- 0.15, N = 3 SE +/- 0.11, N = 3 23.04 23.49 23.97
Apache Spark Row Count: 1000000 - Partitions: 500 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Group By Test Time A B C 0.7358 1.4716 2.2074 2.9432 3.679 SE +/- 0.03, N = 4 SE +/- 0.03, N = 3 3.15 3.20 3.27
Apache Spark Row Count: 10000000 - Partitions: 100 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Inner Join Test Time C B A 3 6 9 12 15 SE +/- 0.27, N = 3 SE +/- 0.02, N = 3 10.93 11.01 11.34
Apache Spark Row Count: 20000000 - Partitions: 2000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Broadcast Inner Join Test Time C B A 5 10 15 20 25 SE +/- 0.18, N = 3 SE +/- 0.31, N = 3 19.31 19.87 20.01
Apache Spark Row Count: 10000000 - Partitions: 500 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Inner Join Test Time C A B 3 6 9 12 15 SE +/- 0.11, N = 3 SE +/- 0.19, N = 3 10.14 10.32 10.48
Apache Spark Row Count: 20000000 - Partitions: 500 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Repartition Test Time A B C 4 8 12 16 20 SE +/- 0.11, N = 3 SE +/- 0.07, N = 4 16.94 17.02 17.49
Apache Spark Row Count: 1000000 - Partitions: 2000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Repartition Test Time B A C 0.5715 1.143 1.7145 2.286 2.8575 SE +/- 0.05, N = 3 SE +/- 0.01, N = 3 2.46 2.50 2.54
Apache Spark Row Count: 1000000 - Partitions: 100 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Repartition Test Time A B C 0.441 0.882 1.323 1.764 2.205 SE +/- 0.01, N = 12 SE +/- 0.03, N = 3 1.90 1.93 1.96
Apache Spark Row Count: 40000000 - Partitions: 100 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Broadcast Inner Join Test Time C B A 9 18 27 36 45 SE +/- 0.23, N = 3 SE +/- 0.51, N = 3 39.62 40.14 40.78
Apache Spark Row Count: 1000000 - Partitions: 2000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Group By Test Time A C B 0.864 1.728 2.592 3.456 4.32 SE +/- 0.04, N = 3 SE +/- 0.04, N = 3 3.74 3.79 3.84
Apache Spark Row Count: 40000000 - Partitions: 500 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - SHA-512 Benchmark Time C B A 11 22 33 44 55 SE +/- 0.23, N = 3 SE +/- 0.32, N = 3 48.84 49.86 50.13
Apache Spark Row Count: 20000000 - Partitions: 1000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Inner Join Test Time A B C 5 10 15 20 25 SE +/- 0.19, N = 3 SE +/- 0.10, N = 3 19.97 20.33 20.49
Apache Spark Row Count: 20000000 - Partitions: 100 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Broadcast Inner Join Test Time A B C 5 10 15 20 25 SE +/- 0.48, N = 3 SE +/- 0.13, N = 3 22.08 22.27 22.65
Apache Spark Row Count: 40000000 - Partitions: 2000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Inner Join Test Time B C A 9 18 27 36 45 SE +/- 0.48, N = 3 SE +/- 0.65, N = 3 39.58 39.85 40.60
Apache Spark Row Count: 10000000 - Partitions: 100 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Broadcast Inner Join Test Time B C A 3 6 9 12 15 SE +/- 0.22, N = 3 SE +/- 0.04, N = 3 11.28 11.54 11.57
Apache Spark Row Count: 1000000 - Partitions: 100 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - SHA-512 Benchmark Time C B A 0.6548 1.3096 1.9644 2.6192 3.274 SE +/- 0.03, N = 3 SE +/- 0.02, N = 12 2.84 2.88 2.91
Apache Spark Row Count: 40000000 - Partitions: 2000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - SHA-512 Benchmark Time C A B 12 24 36 48 60 SE +/- 0.17, N = 3 SE +/- 0.68, N = 3 50.12 50.33 51.35
Apache Spark Row Count: 40000000 - Partitions: 500 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Calculate Pi Benchmark C B A 30 60 90 120 150 SE +/- 0.46, N = 3 SE +/- 2.16, N = 3 147.72 149.13 151.33
Apache Spark Row Count: 10000000 - Partitions: 500 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Group By Test Time C A B 2 4 6 8 10 SE +/- 0.08, N = 3 SE +/- 0.03, N = 3 6.66 6.75 6.82
Apache Spark Row Count: 40000000 - Partitions: 100 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Repartition Test Time A B C 8 16 24 32 40 SE +/- 0.38, N = 3 SE +/- 0.20, N = 3 33.93 34.41 34.74
Apache Spark Row Count: 10000000 - Partitions: 100 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Repartition Test Time C B A 3 6 9 12 15 SE +/- 0.11, N = 3 SE +/- 0.12, N = 3 8.95 9.15 9.16
Apache Spark Row Count: 40000000 - Partitions: 1000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Inner Join Test Time A C B 9 18 27 36 45 SE +/- 0.57, N = 3 SE +/- 0.28, N = 3 39.60 39.90 40.50
Apache Spark Row Count: 10000000 - Partitions: 500 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Repartition Test Time C A B 3 6 9 12 15 SE +/- 0.03, N = 3 SE +/- 0.12, N = 3 8.83 8.94 9.03
Apache Spark Row Count: 40000000 - Partitions: 1000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Calculate Pi Benchmark B A C 30 60 90 120 150 SE +/- 0.29, N = 3 SE +/- 0.40, N = 3 148.66 149.45 151.94
Apache Spark Row Count: 20000000 - Partitions: 1000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Group By Test Time C B A 3 6 9 12 15 SE +/- 0.04, N = 3 SE +/- 0.09, N = 3 10.03 10.16 10.25
Apache Spark Row Count: 20000000 - Partitions: 2000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Repartition Test Time C A B 4 8 12 16 20 SE +/- 0.23, N = 3 SE +/- 0.29, N = 3 17.44 17.52 17.81
Apache Spark Row Count: 1000000 - Partitions: 1000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Group By Test Time C A B 0.7718 1.5436 2.3154 3.0872 3.859 SE +/- 0.05, N = 3 SE +/- 0.02, N = 12 3.36 3.42 3.43
Apache Spark Row Count: 40000000 - Partitions: 2000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Broadcast Inner Join Test Time B C A 9 18 27 36 45 SE +/- 0.38, N = 3 SE +/- 0.68, N = 3 40.71 41.18 41.55
Apache Spark Row Count: 20000000 - Partitions: 500 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Broadcast Inner Join Test Time C B A 5 10 15 20 25 SE +/- 0.22, N = 4 SE +/- 0.11, N = 3 19.94 20.05 20.35
Apache Spark Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark C B A 30 60 90 120 150 SE +/- 0.46, N = 3 SE +/- 0.20, N = 12 146.52 148.74 149.35
Apache Spark Row Count: 1000000 - Partitions: 1000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Repartition Test Time C A B 0.495 0.99 1.485 1.98 2.475 SE +/- 0.01, N = 3 SE +/- 0.01, N = 12 2.16 2.20 2.20
Apache Spark Row Count: 10000000 - Partitions: 2000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Broadcast Inner Join Test Time C B A 3 6 9 12 15 SE +/- 0.15, N = 3 SE +/- 0.31, N = 3 10.41 10.56 10.60
Apache Spark Row Count: 10000000 - Partitions: 1000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Group By Test Time B C A 2 4 6 8 10 SE +/- 0.09, N = 3 SE +/- 0.05, N = 3 6.77 6.86 6.89
Apache Spark Row Count: 40000000 - Partitions: 2000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Repartition Test Time C B A 8 16 24 32 40 SE +/- 0.42, N = 3 SE +/- 0.21, N = 3 33.96 33.98 34.56
Apache Spark Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe C B A 2 4 6 8 10 SE +/- 0.02, N = 3 SE +/- 0.01, N = 12 8.52 8.61 8.67
Apache Spark Row Count: 10000000 - Partitions: 1000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Repartition Test Time C B A 3 6 9 12 15 SE +/- 0.12, N = 3 SE +/- 0.08, N = 3 8.90 9.01 9.05
Apache Spark Row Count: 40000000 - Partitions: 500 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Repartition Test Time A B C 8 16 24 32 40 SE +/- 0.37, N = 3 SE +/- 0.28, N = 3 33.55 33.85 34.11
Apache Spark Row Count: 1000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe C B A 2 4 6 8 10 SE +/- 0.04, N = 3 SE +/- 0.04, N = 4 8.55 8.67 8.69
Apache Spark Row Count: 1000000 - Partitions: 1000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - SHA-512 Benchmark Time C A B 0.7178 1.4356 2.1534 2.8712 3.589 SE +/- 0.02, N = 3 SE +/- 0.03, N = 12 3.14 3.17 3.19
Apache Spark Row Count: 40000000 - Partitions: 500 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Inner Join Test Time A B C 9 18 27 36 45 SE +/- 0.60, N = 3 SE +/- 0.14, N = 3 39.55 40.14 40.17
Apache Spark Row Count: 10000000 - Partitions: 2000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Group By Test Time A C B 2 4 6 8 10 SE +/- 0.08, N = 3 SE +/- 0.02, N = 3 7.03 7.04 7.14
Apache Spark Row Count: 20000000 - Partitions: 2000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Group By Test Time A B C 3 6 9 12 15 SE +/- 0.08, N = 3 SE +/- 0.04, N = 3 10.25 10.32 10.41
Apache Spark Row Count: 40000000 - Partitions: 1000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Repartition Test Time B A C 8 16 24 32 40 SE +/- 0.43, N = 3 SE +/- 0.28, N = 3 34.09 34.50 34.62
Apache Spark Row Count: 40000000 - Partitions: 1000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Group By Test Time A C B 6 12 18 24 30 SE +/- 0.20, N = 3 SE +/- 0.23, N = 3 22.89 23.18 23.24
Apache Spark Row Count: 1000000 - Partitions: 500 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Repartition Test Time A B C 0.4545 0.909 1.3635 1.818 2.2725 SE +/- 0.02, N = 4 SE +/- 0.02, N = 3 1.99 1.99 2.02
Apache Spark Row Count: 20000000 - Partitions: 100 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Group By Test Time B C A 3 6 9 12 15 SE +/- 0.09, N = 3 SE +/- 0.06, N = 3 10.50 10.60 10.65
Apache Spark Row Count: 20000000 - Partitions: 2000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Inner Join Test Time C B A 5 10 15 20 25 SE +/- 0.17, N = 3 SE +/- 0.36, N = 3 20.50 20.72 20.79
Apache Spark Row Count: 10000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.03, N = 3 SE +/- 0.01, N = 3 8.49 8.54 8.61
Apache Spark Row Count: 20000000 - Partitions: 500 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - SHA-512 Benchmark Time C A B 6 12 18 24 30 SE +/- 0.12, N = 3 SE +/- 0.31, N = 4 25.62 25.77 25.98
Apache Spark Row Count: 20000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe B A C 2 4 6 8 10 SE +/- 0.05, N = 3 SE +/- 0.02, N = 3 8.59 8.67 8.71
Apache Spark Row Count: 40000000 - Partitions: 2000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Group By Test Time B C A 6 12 18 24 30 SE +/- 0.07, N = 3 SE +/- 0.21, N = 3 22.94 23.17 23.26
Apache Spark Row Count: 10000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe B A C 2 4 6 8 10 SE +/- 0.034828757, N = 3 SE +/- 0.032542604, N = 3 8.570000000 8.580000000 8.687892725
Apache Spark Row Count: 10000000 - Partitions: 100 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Group By Test Time C A B 2 4 6 8 10 SE +/- 0.04, N = 3 SE +/- 0.07, N = 3 6.55 6.60 6.64
Apache Spark Row Count: 20000000 - Partitions: 100 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Repartition Test Time B A C 4 8 12 16 20 SE +/- 0.19, N = 3 SE +/- 0.09, N = 3 17.81 17.91 18.05
Apache Spark Row Count: 1000000 - Partitions: 500 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - SHA-512 Benchmark Time B C A 0.6885 1.377 2.0655 2.754 3.4425 SE +/- 0.02, N = 3 SE +/- 0.03, N = 4 3.02 3.03 3.06
Apache Spark Row Count: 10000000 - Partitions: 2000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Repartition Test Time B C A 3 6 9 12 15 SE +/- 0.03, N = 3 SE +/- 0.07, N = 3 9.22 9.24 9.34
Apache Spark Row Count: 10000000 - Partitions: 1000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - SHA-512 Benchmark Time B A C 4 8 12 16 20 SE +/- 0.10, N = 3 SE +/- 0.09, N = 3 13.89 13.97 14.07
Apache Spark Row Count: 20000000 - Partitions: 100 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Inner Join Test Time A B C 5 10 15 20 25 SE +/- 0.26, N = 3 SE +/- 0.15, N = 3 21.82 22.08 22.09
Apache Spark Row Count: 40000000 - Partitions: 500 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Broadcast Inner Join Test Time C A B 9 18 27 36 45 SE +/- 0.39, N = 3 SE +/- 0.53, N = 3 40.29 40.32 40.76
Apache Spark Row Count: 10000000 - Partitions: 2000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.22, N = 3 SE +/- 0.57, N = 3 148.55 149.18 150.28
Apache Spark Row Count: 1000000 - Partitions: 500 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Calculate Pi Benchmark C B A 30 60 90 120 150 SE +/- 0.44, N = 3 SE +/- 0.65, N = 4 148.57 149.13 150.18
Apache Spark Row Count: 40000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe C B A 2 4 6 8 10 SE +/- 0.04, N = 3 SE +/- 0.03, N = 3 8.50 8.57 8.59
Apache Spark Row Count: 1000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe C B A 2 4 6 8 10 SE +/- 0.02, N = 12 SE +/- 0.04, N = 3 8.59 8.61 8.68
Apache Spark Row Count: 20000000 - Partitions: 2000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Calculate Pi Benchmark B A C 30 60 90 120 150 SE +/- 0.23, N = 3 SE +/- 0.53, N = 3 148.30 148.85 149.85
Apache Spark Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark B C A 30 60 90 120 150 SE +/- 0.29, N = 3 SE +/- 0.94, N = 3 148.58 149.72 150.03
Apache Spark Row Count: 20000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe C B A 2 4 6 8 10 SE +/- 0.03, N = 3 SE +/- 0.01, N = 3 8.54 8.58 8.62
Apache Spark Row Count: 10000000 - Partitions: 500 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - SHA-512 Benchmark Time A B C 4 8 12 16 20 SE +/- 0.02, N = 3 SE +/- 0.02, N = 3 13.94 13.97 14.07
Apache Spark Row Count: 20000000 - Partitions: 1000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Calculate Pi Benchmark B A C 30 60 90 120 150 SE +/- 0.38, N = 3 SE +/- 0.51, N = 3 147.91 149.23 149.29
Apache Spark Row Count: 10000000 - Partitions: 100 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - SHA-512 Benchmark Time C A B 4 8 12 16 20 SE +/- 0.10, N = 3 SE +/- 0.05, N = 3 14.47 14.54 14.60
Apache Spark Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe B C A 2 4 6 8 10 SE +/- 0.02, N = 3 SE +/- 0.02, N = 3 8.61 8.63 8.68
Apache Spark Row Count: 20000000 - Partitions: 500 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Group By Test Time C B A 3 6 9 12 15 SE +/- 0.06, N = 4 SE +/- 0.08, N = 3 10.04 10.05 10.12
Apache Spark Row Count: 40000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe C A B 2 4 6 8 10 SE +/- 0.02, N = 3 SE +/- 0.03, N = 3 8.56 8.61 8.62
Apache Spark Row Count: 40000000 - Partitions: 100 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Calculate Pi Benchmark B A C 30 60 90 120 150 SE +/- 0.80, N = 3 SE +/- 0.58, N = 3 148.60 149.23 149.61
Apache Spark Row Count: 40000000 - Partitions: 1000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - SHA-512 Benchmark Time A C B 11 22 33 44 55 SE +/- 0.04, N = 3 SE +/- 0.28, N = 3 49.63 49.84 49.95
Apache Spark Row Count: 10000000 - Partitions: 500 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Calculate Pi Benchmark C B A 30 60 90 120 150 SE +/- 0.45, N = 3 SE +/- 0.27, N = 3 148.06 148.92 148.98
Apache Spark Row Count: 20000000 - Partitions: 500 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Calculate Pi Benchmark A C B 30 60 90 120 150 SE +/- 0.25, N = 3 SE +/- 0.53, N = 4 148.22 148.94 149.14
Apache Spark Row Count: 1000000 - Partitions: 500 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Inner Join Test Time C A B 0.3668 0.7336 1.1004 1.4672 1.834 SE +/- 0.04, N = 4 SE +/- 0.03, N = 3 1.62 1.63 1.63
Apache Spark Row Count: 1000000 - Partitions: 1000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Calculate Pi Benchmark B C A 30 60 90 120 150 SE +/- 0.25, N = 12 SE +/- 0.39, N = 3 149.52 149.53 150.41
Apache Spark Row Count: 20000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe C B A 2 4 6 8 10 SE +/- 0.04, N = 3 SE +/- 0.02, N = 3 8.59 8.60 8.64
Apache Spark Row Count: 1000000 - Partitions: 2000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - SHA-512 Benchmark Time A B C 0.783 1.566 2.349 3.132 3.915 SE +/- 0.04, N = 3 SE +/- 0.03, N = 3 3.46 3.47 3.48
Apache Spark Row Count: 40000000 - Partitions: 100 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - SHA-512 Benchmark Time B A C 11 22 33 44 55 SE +/- 0.22, N = 3 SE +/- 0.27, N = 3 49.28 49.41 49.53
Apache Spark Row Count: 40000000 - Partitions: 1000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Broadcast Inner Join Test Time A B C 9 18 27 36 45 SE +/- 0.09, N = 3 SE +/- 1.12, N = 3 40.99 41.01 41.20
Apache Spark Row Count: 20000000 - Partitions: 1000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Repartition Test Time C A B 4 8 12 16 20 SE +/- 0.08, N = 3 SE +/- 0.04, N = 3 17.01 17.03 17.09
Apache Spark Row Count: 10000000 - Partitions: 100 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Calculate Pi Benchmark C B A 30 60 90 120 150 SE +/- 0.44, N = 3 SE +/- 0.68, N = 3 149.07 149.56 149.76
Apache Spark Row Count: 40000000 - Partitions: 2000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Calculate Pi Benchmark C B A 30 60 90 120 150 SE +/- 0.45, N = 3 SE +/- 0.43, N = 3 147.98 148.50 148.60
Apache Spark Row Count: 20000000 - Partitions: 100 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Calculate Pi Benchmark B C A 30 60 90 120 150 SE +/- 0.50, N = 3 SE +/- 0.30, N = 3 148.79 149.25 149.36
Apache Spark Row Count: 40000000 - Partitions: 100 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Group By Test Time B A C 6 12 18 24 30 SE +/- 0.17, N = 3 SE +/- 0.13, N = 3 27.28 27.36 27.38
Apache Spark Row Count: 20000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe C A B 2 4 6 8 10 SE +/- 0.02, N = 3 SE +/- 0.03, N = 4 8.57 8.60 8.60
Apache Spark Row Count: 40000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe C A B 2 4 6 8 10 SE +/- 0.03, N = 3 SE +/- 0.05, N = 3 8.58 8.59 8.61
Apache Spark Row Count: 1000000 - Partitions: 100 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Group By Test Time B C A 0.6503 1.3006 1.9509 2.6012 3.2515 SE +/- 0.01, N = 3 SE +/- 0.02, N = 12 2.88 2.88 2.89
Apache Spark Row Count: 10000000 - Partitions: 1000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Calculate Pi Benchmark B A C 30 60 90 120 150 SE +/- 0.82, N = 3 SE +/- 0.48, N = 3 148.53 148.55 148.99
Apache Spark Row Count: 20000000 - Partitions: 1000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - SHA-512 Benchmark Time B A C 6 12 18 24 30 SE +/- 0.08, N = 3 SE +/- 0.03, N = 3 25.53 25.55 25.60
Apache Spark Row Count: 20000000 - Partitions: 100 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - SHA-512 Benchmark Time A B C 6 12 18 24 30 SE +/- 0.12, N = 3 SE +/- 0.12, N = 3 27.22 27.24 27.29
Apache Spark Row Count: 20000000 - Partitions: 2000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - SHA-512 Benchmark Time A C B 6 12 18 24 30 SE +/- 0.02, N = 3 SE +/- 0.02, N = 3 25.70 25.72 25.76
Apache Spark Row Count: 40000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.02, N = 3 SE +/- 0.01, N = 3 8.63 8.63 8.65
Apache Spark Row Count: 10000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe C A B 2 4 6 8 10 SE +/- 0.01, N = 3 SE +/- 0.01, N = 3 8.64 8.66 8.66
Apache Spark Row Count: 10000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe C B A 2 4 6 8 10 SE +/- 0.03, N = 3 SE +/- 0.02, N = 3 8.67 8.68 8.69
Apache Spark Row Count: 10000000 - Partitions: 2000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - SHA-512 Benchmark Time B C A 4 8 12 16 20 SE +/- 0.10, N = 3 SE +/- 0.06, N = 3 14.12 14.14 14.15
Apache Spark Row Count: 10000000 - Partitions: 1000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Broadcast Inner Join Test Time C B A 3 6 9 12 15 SE +/- 0.41, N = 3 SE +/- 0.43, N = 3 10.08 10.60 10.65
Apache Spark Row Count: 10000000 - Partitions: 1000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Inner Join Test Time C A B 3 6 9 12 15 SE +/- 0.41, N = 3 SE +/- 0.17, N = 3 10.51 10.79 11.02
Apache Spark Row Count: 1000000 - Partitions: 1000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Broadcast Inner Join Test Time A B C 0.3915 0.783 1.1745 1.566 1.9575 SE +/- 0.02, N = 3 SE +/- 0.03, N = 12 1.58 1.66 1.74
Apache Spark Row Count: 1000000 - Partitions: 500 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Broadcast Inner Join Test Time C B A 0.333 0.666 0.999 1.332 1.665 SE +/- 0.02, N = 3 SE +/- 0.07, N = 4 1.38 1.45 1.48
Apache Spark Row Count: 1000000 - Partitions: 100 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Broadcast Inner Join Test Time C A B 0.306 0.612 0.918 1.224 1.53 SE +/- 0.02, N = 12 SE +/- 0.07, N = 3 1.30 1.31 1.36
Apache Spark Row Count: 1000000 - Partitions: 100 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Inner Join Test Time B C A 0.3263 0.6526 0.9789 1.3052 1.6315 SE +/- 0.06, N = 3 SE +/- 0.02, N = 12 1.40 1.44 1.45
Phoronix Test Suite v10.8.4