apache spark 5800X3D AMD Ryzen 7 5800X3D 8-Core testing with a ASUS ROG CROSSHAIR VIII HERO (4201 BIOS) and Gigabyte AMD Radeon RX 6600 8GB on Ubuntu 22.04 via the Phoronix Test Suite.
HTML result view exported from: https://openbenchmarking.org/result/2208042-PTS-APACHESP16&rdt&grr .
apache spark 5800X3D Processor Motherboard Chipset Memory Disk Graphics Audio Monitor Network OS Kernel Desktop Display Server OpenGL Vulkan Compiler File-System Screen Resolution A B C AMD Ryzen 7 5800X3D 8-Core @ 3.40GHz (8 Cores / 16 Threads) ASUS ROG CROSSHAIR VIII HERO (4201 BIOS) AMD Starship/Matisse 32GB 1000GB Western Digital WDS100T1X0E-00AFY0 Gigabyte AMD Radeon RX 6600 8GB (2750/875MHz) AMD Navi 21 HDMI Audio ASUS MG28U Realtek RTL8125 2.5GbE + Intel I211 Ubuntu 22.04 5.19.0-rc6-drm-next-28july (x86_64) GNOME Shell 42.2 X Server 1.21.1.3 + Wayland 4.6 Mesa 22.2.0-devel (git-c6ded153d1) (LLVM 12.0.0 DRM 3.48) 1.3.211 GCC 11.2.0 ext4 3840x2160 OpenBenchmarking.org Kernel Details - Transparent Huge Pages: madvise Processor Details - Scaling Governor: acpi-cpufreq schedutil (Boost: Enabled) - CPU Microcode: 0xa20120a Java Details - OpenJDK Runtime Environment (build 11.0.15+10-Ubuntu-0ubuntu0.22.04.1) Python Details - Python 3.10.4 Security Details - itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling + srbds: Not affected + tsx_async_abort: Not affected
apache spark 5800X3D spark: 1000000 - 1000 - Broadcast Inner Join Test Time spark: 1000000 - 1000 - Inner Join Test Time spark: 1000000 - 1000 - Repartition Test Time spark: 1000000 - 1000 - Group By Test Time spark: 1000000 - 1000 - Calculate Pi Benchmark Using Dataframe spark: 1000000 - 1000 - Calculate Pi Benchmark spark: 1000000 - 1000 - SHA-512 Benchmark Time spark: 1000000 - 100 - Broadcast Inner Join Test Time spark: 1000000 - 100 - Inner Join Test Time spark: 1000000 - 100 - Repartition Test Time spark: 1000000 - 100 - Group By Test Time spark: 1000000 - 100 - Calculate Pi Benchmark Using Dataframe spark: 1000000 - 100 - Calculate Pi Benchmark spark: 1000000 - 100 - SHA-512 Benchmark Time spark: 40000000 - 100 - Broadcast Inner Join Test Time spark: 40000000 - 100 - Inner Join Test Time spark: 40000000 - 100 - Repartition Test Time spark: 40000000 - 100 - Group By Test Time spark: 40000000 - 100 - Calculate Pi Benchmark Using Dataframe spark: 40000000 - 100 - Calculate Pi Benchmark spark: 40000000 - 100 - SHA-512 Benchmark Time spark: 40000000 - 2000 - Broadcast Inner Join Test Time spark: 40000000 - 2000 - Inner Join Test Time spark: 40000000 - 2000 - Repartition Test Time spark: 40000000 - 2000 - Group By Test Time spark: 40000000 - 2000 - Calculate Pi Benchmark Using Dataframe spark: 40000000 - 2000 - Calculate Pi Benchmark spark: 40000000 - 2000 - SHA-512 Benchmark Time spark: 40000000 - 1000 - Broadcast Inner Join Test Time spark: 40000000 - 1000 - Inner Join Test Time spark: 40000000 - 1000 - Repartition Test Time spark: 40000000 - 1000 - Group By Test Time spark: 40000000 - 1000 - Calculate Pi Benchmark Using Dataframe spark: 40000000 - 1000 - Calculate Pi Benchmark spark: 40000000 - 1000 - SHA-512 Benchmark Time spark: 40000000 - 500 - Broadcast Inner Join Test Time spark: 40000000 - 500 - Inner Join Test Time spark: 40000000 - 500 - Repartition Test Time spark: 40000000 - 500 - Group By Test Time spark: 40000000 - 500 - Calculate Pi Benchmark Using Dataframe spark: 40000000 - 500 - Calculate Pi Benchmark spark: 40000000 - 500 - SHA-512 Benchmark Time spark: 20000000 - 500 - Broadcast Inner Join Test Time spark: 20000000 - 500 - Inner Join Test Time spark: 20000000 - 500 - Repartition Test Time spark: 20000000 - 500 - Group By Test Time spark: 20000000 - 500 - Calculate Pi Benchmark Using Dataframe spark: 20000000 - 500 - Calculate Pi Benchmark spark: 20000000 - 500 - SHA-512 Benchmark Time spark: 20000000 - 100 - Broadcast Inner Join Test Time spark: 20000000 - 100 - Inner Join Test Time spark: 20000000 - 100 - Repartition Test Time spark: 20000000 - 100 - Group By Test Time spark: 20000000 - 100 - Calculate Pi Benchmark Using Dataframe spark: 20000000 - 100 - Calculate Pi Benchmark spark: 20000000 - 100 - SHA-512 Benchmark Time spark: 20000000 - 2000 - Broadcast Inner Join Test Time spark: 20000000 - 2000 - Inner Join Test Time spark: 20000000 - 2000 - Repartition Test Time spark: 20000000 - 2000 - Group By Test Time spark: 20000000 - 2000 - Calculate Pi Benchmark Using Dataframe spark: 20000000 - 2000 - Calculate Pi Benchmark spark: 20000000 - 2000 - SHA-512 Benchmark Time spark: 20000000 - 1000 - Broadcast Inner Join Test Time spark: 20000000 - 1000 - Inner Join Test Time spark: 20000000 - 1000 - Repartition Test Time spark: 20000000 - 1000 - Group By Test Time spark: 20000000 - 1000 - Calculate Pi Benchmark Using Dataframe spark: 20000000 - 1000 - Calculate Pi Benchmark spark: 20000000 - 1000 - SHA-512 Benchmark Time spark: 10000000 - 100 - Broadcast Inner Join Test Time spark: 10000000 - 100 - Inner Join Test Time spark: 10000000 - 100 - Repartition Test Time spark: 10000000 - 100 - Group By Test Time spark: 10000000 - 100 - Calculate Pi Benchmark Using Dataframe spark: 10000000 - 100 - Calculate Pi Benchmark spark: 10000000 - 100 - SHA-512 Benchmark Time spark: 10000000 - 2000 - Broadcast Inner Join Test Time spark: 10000000 - 2000 - Inner Join Test Time spark: 10000000 - 2000 - Repartition Test Time spark: 10000000 - 2000 - Group By Test Time spark: 10000000 - 2000 - Calculate Pi Benchmark Using Dataframe spark: 10000000 - 2000 - Calculate Pi Benchmark spark: 10000000 - 2000 - SHA-512 Benchmark Time spark: 10000000 - 1000 - Broadcast Inner Join Test Time spark: 10000000 - 1000 - Inner Join Test Time spark: 10000000 - 1000 - Repartition Test Time spark: 10000000 - 1000 - Group By Test Time spark: 10000000 - 1000 - Calculate Pi Benchmark Using Dataframe spark: 10000000 - 1000 - Calculate Pi Benchmark spark: 10000000 - 1000 - SHA-512 Benchmark Time spark: 10000000 - 500 - Broadcast Inner Join Test Time spark: 10000000 - 500 - Inner Join Test Time spark: 10000000 - 500 - Repartition Test Time spark: 10000000 - 500 - Group By Test Time spark: 10000000 - 500 - Calculate Pi Benchmark Using Dataframe spark: 10000000 - 500 - Calculate Pi Benchmark spark: 10000000 - 500 - SHA-512 Benchmark Time spark: 1000000 - 500 - Broadcast Inner Join Test Time spark: 1000000 - 500 - Inner Join Test Time spark: 1000000 - 500 - Repartition Test Time spark: 1000000 - 500 - Group By Test Time spark: 1000000 - 500 - Calculate Pi Benchmark Using Dataframe spark: 1000000 - 500 - Calculate Pi Benchmark spark: 1000000 - 500 - SHA-512 Benchmark Time spark: 1000000 - 2000 - Broadcast Inner Join Test Time spark: 1000000 - 2000 - Inner Join Test Time spark: 1000000 - 2000 - Repartition Test Time spark: 1000000 - 2000 - Group By Test Time spark: 1000000 - 2000 - Calculate Pi Benchmark Using Dataframe spark: 1000000 - 2000 - Calculate Pi Benchmark spark: 1000000 - 2000 - SHA-512 Benchmark Time A B C 1.58 1.92 2.20 3.42 8.68 150.409490333 3.17 1.31 1.45 1.90 2.89 8.67 149.348708414 2.91 40.78 40.00 33.933058393 27.362051435 8.63 149.227508787 49.41 41.55 40.60 34.56 23.26 8.59 148.601893741 50.33 40.993090142 39.60 34.50 22.89 8.61 149.450658423 49.63 40.32 39.55 33.546966500 23.97 8.59 151.330983793 50.13 20.35 20.035527670 16.94 10.118608339 8.60 148.224458268 25.771477402 22.08 21.82 17.908074200 10.65 8.67 149.362186051 27.22 20.01 20.79 17.52 10.25 8.64 148.853580559 25.70 19.87 19.97 17.03 10.25 8.62 149.230168484 25.55 11.566249356 11.340071822 9.16 6.60 8.66 149.758709975 14.539789898 10.60 11.20 9.34 7.03 8.58 148.548505705 14.15 10.65 10.79 9.05 6.89 8.49 148.548472344 13.97 10.589748727 10.320102526 8.94 6.75 8.69 148.982085231 13.94 1.48 1.63 1.99 3.15 8.69 150.184683924 3.06 2.02 2.32 2.50 3.74 8.68 150.031645092 3.46 1.66 1.84 2.20 3.43 8.61 149.520497364 3.19 1.36 1.40 1.93 2.88 8.61 148.735383131 2.88 40.14 39.87 34.41 27.28 8.63 148.597175671 49.28 40.71 39.58 33.98 22.94 8.57 148.502716195 51.35 41.01 40.50 34.09 23.24 8.62 148.664434137 49.95 40.760680072 40.14 33.85 23.49 8.61 149.128695060 49.86 20.05 19.77 17.02 10.05 8.60 149.139980735 25.98 22.27 22.08 17.81 10.50 8.59 148.786767111 27.24 19.87 20.72 17.81 10.32 8.60 148.300590012 25.76 20.66 20.33 17.09 10.16 8.58 147.914605982 25.53 11.28 11.01 9.15 6.64 8.66 149.564826029 14.60 10.56 11.21 9.22 7.14 8.57 149.183387153 14.12 10.60 11.02 9.01 6.77 8.54 148.532731450 13.89 10.11 10.48 9.03 6.82 8.68 148.915757535 13.97 1.45 1.63 1.99 3.20 8.67 149.134613280 3.02 2.05 2.36 2.46 3.84 8.61 148.583093415 3.47 1.74 1.76 2.16 3.36 8.59 149.529773024 3.14 1.30 1.44 1.96 2.88 8.52 146.52097559 2.84 39.621362405 41.98 34.74 27.382169172 8.65 149.610888179 49.53 41.18 39.848198836 33.96 23.170788559 8.50 147.981380284 50.12 41.20 39.899100315 34.62 23.18 8.56 151.941134869 49.84 40.291389364 40.17 34.11 23.04 8.58 147.715180363 48.84 19.94 21.21 17.493210745 10.04 8.57 148.939367323 25.62 22.650420462 22.09 18.05 10.60 8.71 149.245823899 27.29 19.31 20.50 17.44 10.41 8.59 149.847002868 25.72 21.08 20.49 17.01 10.03 8.54 149.292751295 25.60 11.54 10.93 8.95 6.55 8.64 149.06749502 14.47 10.411906333 11.66 9.24 7.04 8.687892725 150.275810804 14.14 10.08 10.51 8.90 6.86 8.61 148.988566083 14.07 10.24 10.14 8.83 6.66 8.67 148.055891825 14.07 1.38 1.62 2.02 3.27 8.55 148.565766046 3.03 1.94 2.50 2.54 3.79 8.63 149.71767697 3.48 OpenBenchmarking.org
Apache Spark Row Count: 1000000 - Partitions: 1000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Broadcast Inner Join Test Time A B C 0.3915 0.783 1.1745 1.566 1.9575 SE +/- 0.02, N = 3 SE +/- 0.03, N = 12 1.58 1.66 1.74
Apache Spark Row Count: 1000000 - Partitions: 1000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Inner Join Test Time A B C 0.432 0.864 1.296 1.728 2.16 SE +/- 0.01, N = 3 SE +/- 0.02, N = 12 1.92 1.84 1.76
Apache Spark Row Count: 1000000 - Partitions: 1000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Repartition Test Time A B C 0.495 0.99 1.485 1.98 2.475 SE +/- 0.01, N = 3 SE +/- 0.01, N = 12 2.20 2.20 2.16
Apache Spark Row Count: 1000000 - Partitions: 1000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Group By Test Time A B C 0.7718 1.5436 2.3154 3.0872 3.859 SE +/- 0.05, N = 3 SE +/- 0.02, N = 12 3.42 3.43 3.36
Apache Spark Row Count: 1000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.04, N = 3 SE +/- 0.02, N = 12 8.68 8.61 8.59
Apache Spark Row Count: 1000000 - Partitions: 1000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.39, N = 3 SE +/- 0.25, N = 12 150.41 149.52 149.53
Apache Spark Row Count: 1000000 - Partitions: 1000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - SHA-512 Benchmark Time A B C 0.7178 1.4356 2.1534 2.8712 3.589 SE +/- 0.02, N = 3 SE +/- 0.03, N = 12 3.17 3.19 3.14
Apache Spark Row Count: 1000000 - Partitions: 100 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Broadcast Inner Join Test Time A B C 0.306 0.612 0.918 1.224 1.53 SE +/- 0.02, N = 12 SE +/- 0.07, N = 3 1.31 1.36 1.30
Apache Spark Row Count: 1000000 - Partitions: 100 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Inner Join Test Time A B C 0.3263 0.6526 0.9789 1.3052 1.6315 SE +/- 0.02, N = 12 SE +/- 0.06, N = 3 1.45 1.40 1.44
Apache Spark Row Count: 1000000 - Partitions: 100 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Repartition Test Time A B C 0.441 0.882 1.323 1.764 2.205 SE +/- 0.01, N = 12 SE +/- 0.03, N = 3 1.90 1.93 1.96
Apache Spark Row Count: 1000000 - Partitions: 100 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Group By Test Time A B C 0.6503 1.3006 1.9509 2.6012 3.2515 SE +/- 0.02, N = 12 SE +/- 0.01, N = 3 2.89 2.88 2.88
Apache Spark Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.01, N = 12 SE +/- 0.02, N = 3 8.67 8.61 8.52
Apache Spark Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.20, N = 12 SE +/- 0.46, N = 3 149.35 148.74 146.52
Apache Spark Row Count: 1000000 - Partitions: 100 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - SHA-512 Benchmark Time A B C 0.6548 1.3096 1.9644 2.6192 3.274 SE +/- 0.02, N = 12 SE +/- 0.03, N = 3 2.91 2.88 2.84
Apache Spark Row Count: 40000000 - Partitions: 100 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Broadcast Inner Join Test Time A B C 9 18 27 36 45 SE +/- 0.51, N = 3 SE +/- 0.23, N = 3 40.78 40.14 39.62
Apache Spark Row Count: 40000000 - Partitions: 100 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Inner Join Test Time A B C 10 20 30 40 50 SE +/- 0.25, N = 3 SE +/- 0.42, N = 3 40.00 39.87 41.98
Apache Spark Row Count: 40000000 - Partitions: 100 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Repartition Test Time A B C 8 16 24 32 40 SE +/- 0.38, N = 3 SE +/- 0.20, N = 3 33.93 34.41 34.74
Apache Spark Row Count: 40000000 - Partitions: 100 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Group By Test Time A B C 6 12 18 24 30 SE +/- 0.13, N = 3 SE +/- 0.17, N = 3 27.36 27.28 27.38
Apache Spark Row Count: 40000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.02, N = 3 SE +/- 0.01, N = 3 8.63 8.63 8.65
Apache Spark Row Count: 40000000 - Partitions: 100 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.58, N = 3 SE +/- 0.80, N = 3 149.23 148.60 149.61
Apache Spark Row Count: 40000000 - Partitions: 100 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - SHA-512 Benchmark Time A B C 11 22 33 44 55 SE +/- 0.27, N = 3 SE +/- 0.22, N = 3 49.41 49.28 49.53
Apache Spark Row Count: 40000000 - Partitions: 2000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Broadcast Inner Join Test Time A B C 9 18 27 36 45 SE +/- 0.68, N = 3 SE +/- 0.38, N = 3 41.55 40.71 41.18
Apache Spark Row Count: 40000000 - Partitions: 2000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Inner Join Test Time A B C 9 18 27 36 45 SE +/- 0.65, N = 3 SE +/- 0.48, N = 3 40.60 39.58 39.85
Apache Spark Row Count: 40000000 - Partitions: 2000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Repartition Test Time A B C 8 16 24 32 40 SE +/- 0.21, N = 3 SE +/- 0.42, N = 3 34.56 33.98 33.96
Apache Spark Row Count: 40000000 - Partitions: 2000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Group By Test Time A B C 6 12 18 24 30 SE +/- 0.21, N = 3 SE +/- 0.07, N = 3 23.26 22.94 23.17
Apache Spark Row Count: 40000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.03, N = 3 SE +/- 0.04, N = 3 8.59 8.57 8.50
Apache Spark Row Count: 40000000 - Partitions: 2000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.43, N = 3 SE +/- 0.45, N = 3 148.60 148.50 147.98
Apache Spark Row Count: 40000000 - Partitions: 2000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - SHA-512 Benchmark Time A B C 12 24 36 48 60 SE +/- 0.17, N = 3 SE +/- 0.68, N = 3 50.33 51.35 50.12
Apache Spark Row Count: 40000000 - Partitions: 1000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Broadcast Inner Join Test Time A B C 9 18 27 36 45 SE +/- 0.09, N = 3 SE +/- 1.12, N = 3 40.99 41.01 41.20
Apache Spark Row Count: 40000000 - Partitions: 1000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Inner Join Test Time A B C 9 18 27 36 45 SE +/- 0.57, N = 3 SE +/- 0.28, N = 3 39.60 40.50 39.90
Apache Spark Row Count: 40000000 - Partitions: 1000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Repartition Test Time A B C 8 16 24 32 40 SE +/- 0.28, N = 3 SE +/- 0.43, N = 3 34.50 34.09 34.62
Apache Spark Row Count: 40000000 - Partitions: 1000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Group By Test Time A B C 6 12 18 24 30 SE +/- 0.20, N = 3 SE +/- 0.23, N = 3 22.89 23.24 23.18
Apache Spark Row Count: 40000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.02, N = 3 SE +/- 0.03, N = 3 8.61 8.62 8.56
Apache Spark Row Count: 40000000 - Partitions: 1000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.40, N = 3 SE +/- 0.29, N = 3 149.45 148.66 151.94
Apache Spark Row Count: 40000000 - Partitions: 1000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - SHA-512 Benchmark Time A B C 11 22 33 44 55 SE +/- 0.04, N = 3 SE +/- 0.28, N = 3 49.63 49.95 49.84
Apache Spark Row Count: 40000000 - Partitions: 500 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Broadcast Inner Join Test Time A B C 9 18 27 36 45 SE +/- 0.39, N = 3 SE +/- 0.53, N = 3 40.32 40.76 40.29
Apache Spark Row Count: 40000000 - Partitions: 500 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Inner Join Test Time A B C 9 18 27 36 45 SE +/- 0.60, N = 3 SE +/- 0.14, N = 3 39.55 40.14 40.17
Apache Spark Row Count: 40000000 - Partitions: 500 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Repartition Test Time A B C 8 16 24 32 40 SE +/- 0.37, N = 3 SE +/- 0.28, N = 3 33.55 33.85 34.11
Apache Spark Row Count: 40000000 - Partitions: 500 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Group By Test Time A B C 6 12 18 24 30 SE +/- 0.11, N = 3 SE +/- 0.15, N = 3 23.97 23.49 23.04
Apache Spark Row Count: 40000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.03, N = 3 SE +/- 0.05, N = 3 8.59 8.61 8.58
Apache Spark Row Count: 40000000 - Partitions: 500 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 2.16, N = 3 SE +/- 0.46, N = 3 151.33 149.13 147.72
Apache Spark Row Count: 40000000 - Partitions: 500 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - SHA-512 Benchmark Time A B C 11 22 33 44 55 SE +/- 0.32, N = 3 SE +/- 0.23, N = 3 50.13 49.86 48.84
Apache Spark Row Count: 20000000 - Partitions: 500 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Broadcast Inner Join Test Time A B C 5 10 15 20 25 SE +/- 0.11, N = 3 SE +/- 0.22, N = 4 20.35 20.05 19.94
Apache Spark Row Count: 20000000 - Partitions: 500 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Inner Join Test Time A B C 5 10 15 20 25 SE +/- 0.21, N = 3 SE +/- 0.18, N = 4 20.04 19.77 21.21
Apache Spark Row Count: 20000000 - Partitions: 500 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Repartition Test Time A B C 4 8 12 16 20 SE +/- 0.11, N = 3 SE +/- 0.07, N = 4 16.94 17.02 17.49
Apache Spark Row Count: 20000000 - Partitions: 500 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Group By Test Time A B C 3 6 9 12 15 SE +/- 0.08, N = 3 SE +/- 0.06, N = 4 10.12 10.05 10.04
Apache Spark Row Count: 20000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.02, N = 3 SE +/- 0.03, N = 4 8.60 8.60 8.57
Apache Spark Row Count: 20000000 - Partitions: 500 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.25, N = 3 SE +/- 0.53, N = 4 148.22 149.14 148.94
Apache Spark Row Count: 20000000 - Partitions: 500 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - SHA-512 Benchmark Time A B C 6 12 18 24 30 SE +/- 0.12, N = 3 SE +/- 0.31, N = 4 25.77 25.98 25.62
Apache Spark Row Count: 20000000 - Partitions: 100 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Broadcast Inner Join Test Time A B C 5 10 15 20 25 SE +/- 0.48, N = 3 SE +/- 0.13, N = 3 22.08 22.27 22.65
Apache Spark Row Count: 20000000 - Partitions: 100 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Inner Join Test Time A B C 5 10 15 20 25 SE +/- 0.26, N = 3 SE +/- 0.15, N = 3 21.82 22.08 22.09
Apache Spark Row Count: 20000000 - Partitions: 100 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Repartition Test Time A B C 4 8 12 16 20 SE +/- 0.09, N = 3 SE +/- 0.19, N = 3 17.91 17.81 18.05
Apache Spark Row Count: 20000000 - Partitions: 100 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Group By Test Time A B C 3 6 9 12 15 SE +/- 0.06, N = 3 SE +/- 0.09, N = 3 10.65 10.50 10.60
Apache Spark Row Count: 20000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.02, N = 3 SE +/- 0.05, N = 3 8.67 8.59 8.71
Apache Spark Row Count: 20000000 - Partitions: 100 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.30, N = 3 SE +/- 0.50, N = 3 149.36 148.79 149.25
Apache Spark Row Count: 20000000 - Partitions: 100 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - SHA-512 Benchmark Time A B C 6 12 18 24 30 SE +/- 0.12, N = 3 SE +/- 0.12, N = 3 27.22 27.24 27.29
Apache Spark Row Count: 20000000 - Partitions: 2000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Broadcast Inner Join Test Time A B C 5 10 15 20 25 SE +/- 0.31, N = 3 SE +/- 0.18, N = 3 20.01 19.87 19.31
Apache Spark Row Count: 20000000 - Partitions: 2000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Inner Join Test Time A B C 5 10 15 20 25 SE +/- 0.36, N = 3 SE +/- 0.17, N = 3 20.79 20.72 20.50
Apache Spark Row Count: 20000000 - Partitions: 2000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Repartition Test Time A B C 4 8 12 16 20 SE +/- 0.23, N = 3 SE +/- 0.29, N = 3 17.52 17.81 17.44
Apache Spark Row Count: 20000000 - Partitions: 2000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Group By Test Time A B C 3 6 9 12 15 SE +/- 0.08, N = 3 SE +/- 0.04, N = 3 10.25 10.32 10.41
Apache Spark Row Count: 20000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.02, N = 3 SE +/- 0.04, N = 3 8.64 8.60 8.59
Apache Spark Row Count: 20000000 - Partitions: 2000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.53, N = 3 SE +/- 0.23, N = 3 148.85 148.30 149.85
Apache Spark Row Count: 20000000 - Partitions: 2000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - SHA-512 Benchmark Time A B C 6 12 18 24 30 SE +/- 0.02, N = 3 SE +/- 0.02, N = 3 25.70 25.76 25.72
Apache Spark Row Count: 20000000 - Partitions: 1000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Broadcast Inner Join Test Time A B C 5 10 15 20 25 SE +/- 0.23, N = 3 SE +/- 0.37, N = 3 19.87 20.66 21.08
Apache Spark Row Count: 20000000 - Partitions: 1000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Inner Join Test Time A B C 5 10 15 20 25 SE +/- 0.19, N = 3 SE +/- 0.10, N = 3 19.97 20.33 20.49
Apache Spark Row Count: 20000000 - Partitions: 1000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Repartition Test Time A B C 4 8 12 16 20 SE +/- 0.08, N = 3 SE +/- 0.04, N = 3 17.03 17.09 17.01
Apache Spark Row Count: 20000000 - Partitions: 1000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Group By Test Time A B C 3 6 9 12 15 SE +/- 0.09, N = 3 SE +/- 0.04, N = 3 10.25 10.16 10.03
Apache Spark Row Count: 20000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.01, N = 3 SE +/- 0.03, N = 3 8.62 8.58 8.54
Apache Spark Row Count: 20000000 - Partitions: 1000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.51, N = 3 SE +/- 0.38, N = 3 149.23 147.91 149.29
Apache Spark Row Count: 20000000 - Partitions: 1000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - SHA-512 Benchmark Time A B C 6 12 18 24 30 SE +/- 0.03, N = 3 SE +/- 0.08, N = 3 25.55 25.53 25.60
Apache Spark Row Count: 10000000 - Partitions: 100 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Broadcast Inner Join Test Time A B C 3 6 9 12 15 SE +/- 0.04, N = 3 SE +/- 0.22, N = 3 11.57 11.28 11.54
Apache Spark Row Count: 10000000 - Partitions: 100 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Inner Join Test Time A B C 3 6 9 12 15 SE +/- 0.02, N = 3 SE +/- 0.27, N = 3 11.34 11.01 10.93
Apache Spark Row Count: 10000000 - Partitions: 100 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Repartition Test Time A B C 3 6 9 12 15 SE +/- 0.12, N = 3 SE +/- 0.11, N = 3 9.16 9.15 8.95
Apache Spark Row Count: 10000000 - Partitions: 100 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Group By Test Time A B C 2 4 6 8 10 SE +/- 0.04, N = 3 SE +/- 0.07, N = 3 6.60 6.64 6.55
Apache Spark Row Count: 10000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.01, N = 3 SE +/- 0.01, N = 3 8.66 8.66 8.64
Apache Spark Row Count: 10000000 - Partitions: 100 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.68, N = 3 SE +/- 0.44, N = 3 149.76 149.56 149.07
Apache Spark Row Count: 10000000 - Partitions: 100 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - SHA-512 Benchmark Time A B C 4 8 12 16 20 SE +/- 0.10, N = 3 SE +/- 0.05, N = 3 14.54 14.60 14.47
Apache Spark Row Count: 10000000 - Partitions: 2000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Broadcast Inner Join Test Time A B C 3 6 9 12 15 SE +/- 0.31, N = 3 SE +/- 0.15, N = 3 10.60 10.56 10.41
Apache Spark Row Count: 10000000 - Partitions: 2000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Inner Join Test Time A B C 3 6 9 12 15 SE +/- 0.18, N = 3 SE +/- 0.16, N = 3 11.20 11.21 11.66
Apache Spark Row Count: 10000000 - Partitions: 2000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Repartition Test Time A B C 3 6 9 12 15 SE +/- 0.07, N = 3 SE +/- 0.03, N = 3 9.34 9.22 9.24
Apache Spark Row Count: 10000000 - Partitions: 2000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Group By Test Time A B C 2 4 6 8 10 SE +/- 0.08, N = 3 SE +/- 0.02, N = 3 7.03 7.14 7.04
Apache Spark Row Count: 10000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.032542604, N = 3 SE +/- 0.034828757, N = 3 8.580000000 8.570000000 8.687892725
Apache Spark Row Count: 10000000 - Partitions: 2000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.22, N = 3 SE +/- 0.57, N = 3 148.55 149.18 150.28
Apache Spark Row Count: 10000000 - Partitions: 2000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - SHA-512 Benchmark Time A B C 4 8 12 16 20 SE +/- 0.06, N = 3 SE +/- 0.10, N = 3 14.15 14.12 14.14
Apache Spark Row Count: 10000000 - Partitions: 1000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Broadcast Inner Join Test Time A B C 3 6 9 12 15 SE +/- 0.43, N = 3 SE +/- 0.41, N = 3 10.65 10.60 10.08
Apache Spark Row Count: 10000000 - Partitions: 1000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Inner Join Test Time A B C 3 6 9 12 15 SE +/- 0.41, N = 3 SE +/- 0.17, N = 3 10.79 11.02 10.51
Apache Spark Row Count: 10000000 - Partitions: 1000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Repartition Test Time A B C 3 6 9 12 15 SE +/- 0.08, N = 3 SE +/- 0.12, N = 3 9.05 9.01 8.90
Apache Spark Row Count: 10000000 - Partitions: 1000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Group By Test Time A B C 2 4 6 8 10 SE +/- 0.05, N = 3 SE +/- 0.09, N = 3 6.89 6.77 6.86
Apache Spark Row Count: 10000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.03, N = 3 SE +/- 0.01, N = 3 8.49 8.54 8.61
Apache Spark Row Count: 10000000 - Partitions: 1000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.48, N = 3 SE +/- 0.82, N = 3 148.55 148.53 148.99
Apache Spark Row Count: 10000000 - Partitions: 1000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - SHA-512 Benchmark Time A B C 4 8 12 16 20 SE +/- 0.09, N = 3 SE +/- 0.10, N = 3 13.97 13.89 14.07
Apache Spark Row Count: 10000000 - Partitions: 500 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Broadcast Inner Join Test Time A B C 3 6 9 12 15 SE +/- 0.16, N = 3 SE +/- 0.04, N = 3 10.59 10.11 10.24
Apache Spark Row Count: 10000000 - Partitions: 500 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Inner Join Test Time A B C 3 6 9 12 15 SE +/- 0.11, N = 3 SE +/- 0.19, N = 3 10.32 10.48 10.14
Apache Spark Row Count: 10000000 - Partitions: 500 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Repartition Test Time A B C 3 6 9 12 15 SE +/- 0.03, N = 3 SE +/- 0.12, N = 3 8.94 9.03 8.83
Apache Spark Row Count: 10000000 - Partitions: 500 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Group By Test Time A B C 2 4 6 8 10 SE +/- 0.08, N = 3 SE +/- 0.03, N = 3 6.75 6.82 6.66
Apache Spark Row Count: 10000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.02, N = 3 SE +/- 0.03, N = 3 8.69 8.68 8.67
Apache Spark Row Count: 10000000 - Partitions: 500 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.27, N = 3 SE +/- 0.45, N = 3 148.98 148.92 148.06
Apache Spark Row Count: 10000000 - Partitions: 500 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - SHA-512 Benchmark Time A B C 4 8 12 16 20 SE +/- 0.02, N = 3 SE +/- 0.02, N = 3 13.94 13.97 14.07
Apache Spark Row Count: 1000000 - Partitions: 500 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Broadcast Inner Join Test Time A B C 0.333 0.666 0.999 1.332 1.665 SE +/- 0.07, N = 4 SE +/- 0.02, N = 3 1.48 1.45 1.38
Apache Spark Row Count: 1000000 - Partitions: 500 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Inner Join Test Time A B C 0.3668 0.7336 1.1004 1.4672 1.834 SE +/- 0.04, N = 4 SE +/- 0.03, N = 3 1.63 1.63 1.62
Apache Spark Row Count: 1000000 - Partitions: 500 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Repartition Test Time A B C 0.4545 0.909 1.3635 1.818 2.2725 SE +/- 0.02, N = 4 SE +/- 0.02, N = 3 1.99 1.99 2.02
Apache Spark Row Count: 1000000 - Partitions: 500 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Group By Test Time A B C 0.7358 1.4716 2.2074 2.9432 3.679 SE +/- 0.03, N = 4 SE +/- 0.03, N = 3 3.15 3.20 3.27
Apache Spark Row Count: 1000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.04, N = 4 SE +/- 0.04, N = 3 8.69 8.67 8.55
Apache Spark Row Count: 1000000 - Partitions: 500 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.65, N = 4 SE +/- 0.44, N = 3 150.18 149.13 148.57
Apache Spark Row Count: 1000000 - Partitions: 500 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - SHA-512 Benchmark Time A B C 0.6885 1.377 2.0655 2.754 3.4425 SE +/- 0.03, N = 4 SE +/- 0.02, N = 3 3.06 3.02 3.03
Apache Spark Row Count: 1000000 - Partitions: 2000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Broadcast Inner Join Test Time A B C 0.4613 0.9226 1.3839 1.8452 2.3065 SE +/- 0.04, N = 3 SE +/- 0.05, N = 3 2.02 2.05 1.94
Apache Spark Row Count: 1000000 - Partitions: 2000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Inner Join Test Time A B C 0.5625 1.125 1.6875 2.25 2.8125 SE +/- 0.07, N = 3 SE +/- 0.07, N = 3 2.32 2.36 2.50
Apache Spark Row Count: 1000000 - Partitions: 2000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Repartition Test Time A B C 0.5715 1.143 1.7145 2.286 2.8575 SE +/- 0.01, N = 3 SE +/- 0.05, N = 3 2.50 2.46 2.54
Apache Spark Row Count: 1000000 - Partitions: 2000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Group By Test Time A B C 0.864 1.728 2.592 3.456 4.32 SE +/- 0.04, N = 3 SE +/- 0.04, N = 3 3.74 3.84 3.79
Apache Spark Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.02, N = 3 SE +/- 0.02, N = 3 8.68 8.61 8.63
Apache Spark Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.94, N = 3 SE +/- 0.29, N = 3 150.03 148.58 149.72
Apache Spark Row Count: 1000000 - Partitions: 2000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - SHA-512 Benchmark Time A B C 0.783 1.566 2.349 3.132 3.915 SE +/- 0.04, N = 3 SE +/- 0.03, N = 3 3.46 3.47 3.48
Phoronix Test Suite v10.8.5