apache spark 5800X3D AMD Ryzen 7 5800X3D 8-Core testing with a ASUS ROG CROSSHAIR VIII HERO (4201 BIOS) and Gigabyte AMD Radeon RX 6600 8GB on Ubuntu 22.04 via the Phoronix Test Suite.
HTML result view exported from: https://openbenchmarking.org/result/2208042-PTS-APACHESP16&gru .
apache spark 5800X3D Processor Motherboard Chipset Memory Disk Graphics Audio Monitor Network OS Kernel Desktop Display Server OpenGL Vulkan Compiler File-System Screen Resolution A B C AMD Ryzen 7 5800X3D 8-Core @ 3.40GHz (8 Cores / 16 Threads) ASUS ROG CROSSHAIR VIII HERO (4201 BIOS) AMD Starship/Matisse 32GB 1000GB Western Digital WDS100T1X0E-00AFY0 Gigabyte AMD Radeon RX 6600 8GB (2750/875MHz) AMD Navi 21 HDMI Audio ASUS MG28U Realtek RTL8125 2.5GbE + Intel I211 Ubuntu 22.04 5.19.0-rc6-drm-next-28july (x86_64) GNOME Shell 42.2 X Server 1.21.1.3 + Wayland 4.6 Mesa 22.2.0-devel (git-c6ded153d1) (LLVM 12.0.0 DRM 3.48) 1.3.211 GCC 11.2.0 ext4 3840x2160 OpenBenchmarking.org Kernel Details - Transparent Huge Pages: madvise Processor Details - Scaling Governor: acpi-cpufreq schedutil (Boost: Enabled) - CPU Microcode: 0xa20120a Java Details - OpenJDK Runtime Environment (build 11.0.15+10-Ubuntu-0ubuntu0.22.04.1) Python Details - Python 3.10.4 Security Details - itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling + srbds: Not affected + tsx_async_abort: Not affected
apache spark 5800X3D spark: 1000000 - 100 - SHA-512 Benchmark Time spark: 1000000 - 100 - Calculate Pi Benchmark spark: 1000000 - 100 - Calculate Pi Benchmark Using Dataframe spark: 1000000 - 100 - Group By Test Time spark: 1000000 - 100 - Repartition Test Time spark: 1000000 - 100 - Inner Join Test Time spark: 1000000 - 100 - Broadcast Inner Join Test Time spark: 1000000 - 500 - SHA-512 Benchmark Time spark: 1000000 - 500 - Calculate Pi Benchmark spark: 1000000 - 500 - Calculate Pi Benchmark Using Dataframe spark: 1000000 - 500 - Group By Test Time spark: 1000000 - 500 - Repartition Test Time spark: 1000000 - 500 - Inner Join Test Time spark: 1000000 - 500 - Broadcast Inner Join Test Time spark: 1000000 - 1000 - SHA-512 Benchmark Time spark: 1000000 - 1000 - Calculate Pi Benchmark spark: 1000000 - 1000 - Calculate Pi Benchmark Using Dataframe spark: 1000000 - 1000 - Group By Test Time spark: 1000000 - 1000 - Repartition Test Time spark: 1000000 - 1000 - Inner Join Test Time spark: 1000000 - 1000 - Broadcast Inner Join Test Time spark: 1000000 - 2000 - SHA-512 Benchmark Time spark: 1000000 - 2000 - Calculate Pi Benchmark spark: 1000000 - 2000 - Calculate Pi Benchmark Using Dataframe spark: 1000000 - 2000 - Group By Test Time spark: 1000000 - 2000 - Repartition Test Time spark: 1000000 - 2000 - Inner Join Test Time spark: 1000000 - 2000 - Broadcast Inner Join Test Time spark: 10000000 - 100 - SHA-512 Benchmark Time spark: 10000000 - 100 - Calculate Pi Benchmark spark: 10000000 - 100 - Calculate Pi Benchmark Using Dataframe spark: 10000000 - 100 - Group By Test Time spark: 10000000 - 100 - Repartition Test Time spark: 10000000 - 100 - Inner Join Test Time spark: 10000000 - 100 - Broadcast Inner Join Test Time spark: 10000000 - 500 - SHA-512 Benchmark Time spark: 10000000 - 500 - Calculate Pi Benchmark spark: 10000000 - 500 - Calculate Pi Benchmark Using Dataframe spark: 10000000 - 500 - Group By Test Time spark: 10000000 - 500 - Repartition Test Time spark: 10000000 - 500 - Inner Join Test Time spark: 10000000 - 500 - Broadcast Inner Join Test Time spark: 20000000 - 100 - SHA-512 Benchmark Time spark: 20000000 - 100 - Calculate Pi Benchmark spark: 20000000 - 100 - Calculate Pi Benchmark Using Dataframe spark: 20000000 - 100 - Group By Test Time spark: 20000000 - 100 - Repartition Test Time spark: 20000000 - 100 - Inner Join Test Time spark: 20000000 - 100 - Broadcast Inner Join Test Time spark: 20000000 - 500 - SHA-512 Benchmark Time spark: 20000000 - 500 - Calculate Pi Benchmark spark: 20000000 - 500 - Calculate Pi Benchmark Using Dataframe spark: 20000000 - 500 - Group By Test Time spark: 20000000 - 500 - Repartition Test Time spark: 20000000 - 500 - Inner Join Test Time spark: 20000000 - 500 - Broadcast Inner Join Test Time spark: 40000000 - 100 - SHA-512 Benchmark Time spark: 40000000 - 100 - Calculate Pi Benchmark spark: 40000000 - 100 - Calculate Pi Benchmark Using Dataframe spark: 40000000 - 100 - Group By Test Time spark: 40000000 - 100 - Repartition Test Time spark: 40000000 - 100 - Inner Join Test Time spark: 40000000 - 100 - Broadcast Inner Join Test Time spark: 40000000 - 500 - SHA-512 Benchmark Time spark: 40000000 - 500 - Calculate Pi Benchmark spark: 40000000 - 500 - Calculate Pi Benchmark Using Dataframe spark: 40000000 - 500 - Group By Test Time spark: 40000000 - 500 - Repartition Test Time spark: 40000000 - 500 - Inner Join Test Time spark: 40000000 - 500 - Broadcast Inner Join Test Time spark: 10000000 - 1000 - SHA-512 Benchmark Time spark: 10000000 - 1000 - Calculate Pi Benchmark spark: 10000000 - 1000 - Calculate Pi Benchmark Using Dataframe spark: 10000000 - 1000 - Group By Test Time spark: 10000000 - 1000 - Repartition Test Time spark: 10000000 - 1000 - Inner Join Test Time spark: 10000000 - 1000 - Broadcast Inner Join Test Time spark: 10000000 - 2000 - SHA-512 Benchmark Time spark: 10000000 - 2000 - Calculate Pi Benchmark spark: 10000000 - 2000 - Calculate Pi Benchmark Using Dataframe spark: 10000000 - 2000 - Group By Test Time spark: 10000000 - 2000 - Repartition Test Time spark: 10000000 - 2000 - Inner Join Test Time spark: 10000000 - 2000 - Broadcast Inner Join Test Time spark: 20000000 - 1000 - SHA-512 Benchmark Time spark: 20000000 - 1000 - Calculate Pi Benchmark spark: 20000000 - 1000 - Calculate Pi Benchmark Using Dataframe spark: 20000000 - 1000 - Group By Test Time spark: 20000000 - 1000 - Repartition Test Time spark: 20000000 - 1000 - Inner Join Test Time spark: 20000000 - 1000 - Broadcast Inner Join Test Time spark: 20000000 - 2000 - SHA-512 Benchmark Time spark: 20000000 - 2000 - Calculate Pi Benchmark spark: 20000000 - 2000 - Calculate Pi Benchmark Using Dataframe spark: 20000000 - 2000 - Group By Test Time spark: 20000000 - 2000 - Repartition Test Time spark: 20000000 - 2000 - Inner Join Test Time spark: 20000000 - 2000 - Broadcast Inner Join Test Time spark: 40000000 - 1000 - SHA-512 Benchmark Time spark: 40000000 - 1000 - Calculate Pi Benchmark spark: 40000000 - 1000 - Calculate Pi Benchmark Using Dataframe spark: 40000000 - 1000 - Group By Test Time spark: 40000000 - 1000 - Repartition Test Time spark: 40000000 - 1000 - Inner Join Test Time spark: 40000000 - 1000 - Broadcast Inner Join Test Time spark: 40000000 - 2000 - SHA-512 Benchmark Time spark: 40000000 - 2000 - Calculate Pi Benchmark spark: 40000000 - 2000 - Calculate Pi Benchmark Using Dataframe spark: 40000000 - 2000 - Group By Test Time spark: 40000000 - 2000 - Repartition Test Time spark: 40000000 - 2000 - Inner Join Test Time spark: 40000000 - 2000 - Broadcast Inner Join Test Time A B C 2.91 149.348708414 8.67 2.89 1.90 1.45 1.31 3.06 150.184683924 8.69 3.15 1.99 1.63 1.48 3.17 150.409490333 8.68 3.42 2.20 1.92 1.58 3.46 150.031645092 8.68 3.74 2.50 2.32 2.02 14.539789898 149.758709975 8.66 6.60 9.16 11.340071822 11.566249356 13.94 148.982085231 8.69 6.75 8.94 10.320102526 10.589748727 27.22 149.362186051 8.67 10.65 17.908074200 21.82 22.08 25.771477402 148.224458268 8.60 10.118608339 16.94 20.035527670 20.35 49.41 149.227508787 8.63 27.362051435 33.933058393 40.00 40.78 50.13 151.330983793 8.59 23.97 33.546966500 39.55 40.32 13.97 148.548472344 8.49 6.89 9.05 10.79 10.65 14.15 148.548505705 8.58 7.03 9.34 11.20 10.60 25.55 149.230168484 8.62 10.25 17.03 19.97 19.87 25.70 148.853580559 8.64 10.25 17.52 20.79 20.01 49.63 149.450658423 8.61 22.89 34.50 39.60 40.993090142 50.33 148.601893741 8.59 23.26 34.56 40.60 41.55 2.88 148.735383131 8.61 2.88 1.93 1.40 1.36 3.02 149.134613280 8.67 3.20 1.99 1.63 1.45 3.19 149.520497364 8.61 3.43 2.20 1.84 1.66 3.47 148.583093415 8.61 3.84 2.46 2.36 2.05 14.60 149.564826029 8.66 6.64 9.15 11.01 11.28 13.97 148.915757535 8.68 6.82 9.03 10.48 10.11 27.24 148.786767111 8.59 10.50 17.81 22.08 22.27 25.98 149.139980735 8.60 10.05 17.02 19.77 20.05 49.28 148.597175671 8.63 27.28 34.41 39.87 40.14 49.86 149.128695060 8.61 23.49 33.85 40.14 40.760680072 13.89 148.532731450 8.54 6.77 9.01 11.02 10.60 14.12 149.183387153 8.57 7.14 9.22 11.21 10.56 25.53 147.914605982 8.58 10.16 17.09 20.33 20.66 25.76 148.300590012 8.60 10.32 17.81 20.72 19.87 49.95 148.664434137 8.62 23.24 34.09 40.50 41.01 51.35 148.502716195 8.57 22.94 33.98 39.58 40.71 2.84 146.52097559 8.52 2.88 1.96 1.44 1.30 3.03 148.565766046 8.55 3.27 2.02 1.62 1.38 3.14 149.529773024 8.59 3.36 2.16 1.76 1.74 3.48 149.71767697 8.63 3.79 2.54 2.50 1.94 14.47 149.06749502 8.64 6.55 8.95 10.93 11.54 14.07 148.055891825 8.67 6.66 8.83 10.14 10.24 27.29 149.245823899 8.71 10.60 18.05 22.09 22.650420462 25.62 148.939367323 8.57 10.04 17.493210745 21.21 19.94 49.53 149.610888179 8.65 27.382169172 34.74 41.98 39.621362405 48.84 147.715180363 8.58 23.04 34.11 40.17 40.291389364 14.07 148.988566083 8.61 6.86 8.90 10.51 10.08 14.14 150.275810804 8.687892725 7.04 9.24 11.66 10.411906333 25.60 149.292751295 8.54 10.03 17.01 20.49 21.08 25.72 149.847002868 8.59 10.41 17.44 20.50 19.31 49.84 151.941134869 8.56 23.18 34.62 39.899100315 41.20 50.12 147.981380284 8.50 23.170788559 33.96 39.848198836 41.18 OpenBenchmarking.org
Apache Spark Row Count: 1000000 - Partitions: 100 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - SHA-512 Benchmark Time A B C 0.6548 1.3096 1.9644 2.6192 3.274 SE +/- 0.02, N = 12 SE +/- 0.03, N = 3 2.91 2.88 2.84
Apache Spark Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.20, N = 12 SE +/- 0.46, N = 3 149.35 148.74 146.52
Apache Spark Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.01, N = 12 SE +/- 0.02, N = 3 8.67 8.61 8.52
Apache Spark Row Count: 1000000 - Partitions: 100 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Group By Test Time A B C 0.6503 1.3006 1.9509 2.6012 3.2515 SE +/- 0.02, N = 12 SE +/- 0.01, N = 3 2.89 2.88 2.88
Apache Spark Row Count: 1000000 - Partitions: 100 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Repartition Test Time A B C 0.441 0.882 1.323 1.764 2.205 SE +/- 0.01, N = 12 SE +/- 0.03, N = 3 1.90 1.93 1.96
Apache Spark Row Count: 1000000 - Partitions: 100 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Inner Join Test Time A B C 0.3263 0.6526 0.9789 1.3052 1.6315 SE +/- 0.02, N = 12 SE +/- 0.06, N = 3 1.45 1.40 1.44
Apache Spark Row Count: 1000000 - Partitions: 100 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Broadcast Inner Join Test Time A B C 0.306 0.612 0.918 1.224 1.53 SE +/- 0.02, N = 12 SE +/- 0.07, N = 3 1.31 1.36 1.30
Apache Spark Row Count: 1000000 - Partitions: 500 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - SHA-512 Benchmark Time A B C 0.6885 1.377 2.0655 2.754 3.4425 SE +/- 0.03, N = 4 SE +/- 0.02, N = 3 3.06 3.02 3.03
Apache Spark Row Count: 1000000 - Partitions: 500 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.65, N = 4 SE +/- 0.44, N = 3 150.18 149.13 148.57
Apache Spark Row Count: 1000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.04, N = 4 SE +/- 0.04, N = 3 8.69 8.67 8.55
Apache Spark Row Count: 1000000 - Partitions: 500 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Group By Test Time A B C 0.7358 1.4716 2.2074 2.9432 3.679 SE +/- 0.03, N = 4 SE +/- 0.03, N = 3 3.15 3.20 3.27
Apache Spark Row Count: 1000000 - Partitions: 500 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Repartition Test Time A B C 0.4545 0.909 1.3635 1.818 2.2725 SE +/- 0.02, N = 4 SE +/- 0.02, N = 3 1.99 1.99 2.02
Apache Spark Row Count: 1000000 - Partitions: 500 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Inner Join Test Time A B C 0.3668 0.7336 1.1004 1.4672 1.834 SE +/- 0.04, N = 4 SE +/- 0.03, N = 3 1.63 1.63 1.62
Apache Spark Row Count: 1000000 - Partitions: 500 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Broadcast Inner Join Test Time A B C 0.333 0.666 0.999 1.332 1.665 SE +/- 0.07, N = 4 SE +/- 0.02, N = 3 1.48 1.45 1.38
Apache Spark Row Count: 1000000 - Partitions: 1000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - SHA-512 Benchmark Time A B C 0.7178 1.4356 2.1534 2.8712 3.589 SE +/- 0.02, N = 3 SE +/- 0.03, N = 12 3.17 3.19 3.14
Apache Spark Row Count: 1000000 - Partitions: 1000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.39, N = 3 SE +/- 0.25, N = 12 150.41 149.52 149.53
Apache Spark Row Count: 1000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.04, N = 3 SE +/- 0.02, N = 12 8.68 8.61 8.59
Apache Spark Row Count: 1000000 - Partitions: 1000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Group By Test Time A B C 0.7718 1.5436 2.3154 3.0872 3.859 SE +/- 0.05, N = 3 SE +/- 0.02, N = 12 3.42 3.43 3.36
Apache Spark Row Count: 1000000 - Partitions: 1000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Repartition Test Time A B C 0.495 0.99 1.485 1.98 2.475 SE +/- 0.01, N = 3 SE +/- 0.01, N = 12 2.20 2.20 2.16
Apache Spark Row Count: 1000000 - Partitions: 1000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Inner Join Test Time A B C 0.432 0.864 1.296 1.728 2.16 SE +/- 0.01, N = 3 SE +/- 0.02, N = 12 1.92 1.84 1.76
Apache Spark Row Count: 1000000 - Partitions: 1000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Broadcast Inner Join Test Time A B C 0.3915 0.783 1.1745 1.566 1.9575 SE +/- 0.02, N = 3 SE +/- 0.03, N = 12 1.58 1.66 1.74
Apache Spark Row Count: 1000000 - Partitions: 2000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - SHA-512 Benchmark Time A B C 0.783 1.566 2.349 3.132 3.915 SE +/- 0.04, N = 3 SE +/- 0.03, N = 3 3.46 3.47 3.48
Apache Spark Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.94, N = 3 SE +/- 0.29, N = 3 150.03 148.58 149.72
Apache Spark Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.02, N = 3 SE +/- 0.02, N = 3 8.68 8.61 8.63
Apache Spark Row Count: 1000000 - Partitions: 2000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Group By Test Time A B C 0.864 1.728 2.592 3.456 4.32 SE +/- 0.04, N = 3 SE +/- 0.04, N = 3 3.74 3.84 3.79
Apache Spark Row Count: 1000000 - Partitions: 2000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Repartition Test Time A B C 0.5715 1.143 1.7145 2.286 2.8575 SE +/- 0.01, N = 3 SE +/- 0.05, N = 3 2.50 2.46 2.54
Apache Spark Row Count: 1000000 - Partitions: 2000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Inner Join Test Time A B C 0.5625 1.125 1.6875 2.25 2.8125 SE +/- 0.07, N = 3 SE +/- 0.07, N = 3 2.32 2.36 2.50
Apache Spark Row Count: 1000000 - Partitions: 2000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Broadcast Inner Join Test Time A B C 0.4613 0.9226 1.3839 1.8452 2.3065 SE +/- 0.04, N = 3 SE +/- 0.05, N = 3 2.02 2.05 1.94
Apache Spark Row Count: 10000000 - Partitions: 100 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - SHA-512 Benchmark Time A B C 4 8 12 16 20 SE +/- 0.10, N = 3 SE +/- 0.05, N = 3 14.54 14.60 14.47
Apache Spark Row Count: 10000000 - Partitions: 100 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.68, N = 3 SE +/- 0.44, N = 3 149.76 149.56 149.07
Apache Spark Row Count: 10000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.01, N = 3 SE +/- 0.01, N = 3 8.66 8.66 8.64
Apache Spark Row Count: 10000000 - Partitions: 100 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Group By Test Time A B C 2 4 6 8 10 SE +/- 0.04, N = 3 SE +/- 0.07, N = 3 6.60 6.64 6.55
Apache Spark Row Count: 10000000 - Partitions: 100 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Repartition Test Time A B C 3 6 9 12 15 SE +/- 0.12, N = 3 SE +/- 0.11, N = 3 9.16 9.15 8.95
Apache Spark Row Count: 10000000 - Partitions: 100 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Inner Join Test Time A B C 3 6 9 12 15 SE +/- 0.02, N = 3 SE +/- 0.27, N = 3 11.34 11.01 10.93
Apache Spark Row Count: 10000000 - Partitions: 100 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Broadcast Inner Join Test Time A B C 3 6 9 12 15 SE +/- 0.04, N = 3 SE +/- 0.22, N = 3 11.57 11.28 11.54
Apache Spark Row Count: 10000000 - Partitions: 500 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - SHA-512 Benchmark Time A B C 4 8 12 16 20 SE +/- 0.02, N = 3 SE +/- 0.02, N = 3 13.94 13.97 14.07
Apache Spark Row Count: 10000000 - Partitions: 500 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.27, N = 3 SE +/- 0.45, N = 3 148.98 148.92 148.06
Apache Spark Row Count: 10000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.02, N = 3 SE +/- 0.03, N = 3 8.69 8.68 8.67
Apache Spark Row Count: 10000000 - Partitions: 500 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Group By Test Time A B C 2 4 6 8 10 SE +/- 0.08, N = 3 SE +/- 0.03, N = 3 6.75 6.82 6.66
Apache Spark Row Count: 10000000 - Partitions: 500 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Repartition Test Time A B C 3 6 9 12 15 SE +/- 0.03, N = 3 SE +/- 0.12, N = 3 8.94 9.03 8.83
Apache Spark Row Count: 10000000 - Partitions: 500 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Inner Join Test Time A B C 3 6 9 12 15 SE +/- 0.11, N = 3 SE +/- 0.19, N = 3 10.32 10.48 10.14
Apache Spark Row Count: 10000000 - Partitions: 500 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Broadcast Inner Join Test Time A B C 3 6 9 12 15 SE +/- 0.16, N = 3 SE +/- 0.04, N = 3 10.59 10.11 10.24
Apache Spark Row Count: 20000000 - Partitions: 100 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - SHA-512 Benchmark Time A B C 6 12 18 24 30 SE +/- 0.12, N = 3 SE +/- 0.12, N = 3 27.22 27.24 27.29
Apache Spark Row Count: 20000000 - Partitions: 100 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.30, N = 3 SE +/- 0.50, N = 3 149.36 148.79 149.25
Apache Spark Row Count: 20000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.02, N = 3 SE +/- 0.05, N = 3 8.67 8.59 8.71
Apache Spark Row Count: 20000000 - Partitions: 100 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Group By Test Time A B C 3 6 9 12 15 SE +/- 0.06, N = 3 SE +/- 0.09, N = 3 10.65 10.50 10.60
Apache Spark Row Count: 20000000 - Partitions: 100 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Repartition Test Time A B C 4 8 12 16 20 SE +/- 0.09, N = 3 SE +/- 0.19, N = 3 17.91 17.81 18.05
Apache Spark Row Count: 20000000 - Partitions: 100 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Inner Join Test Time A B C 5 10 15 20 25 SE +/- 0.26, N = 3 SE +/- 0.15, N = 3 21.82 22.08 22.09
Apache Spark Row Count: 20000000 - Partitions: 100 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Broadcast Inner Join Test Time A B C 5 10 15 20 25 SE +/- 0.48, N = 3 SE +/- 0.13, N = 3 22.08 22.27 22.65
Apache Spark Row Count: 20000000 - Partitions: 500 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - SHA-512 Benchmark Time A B C 6 12 18 24 30 SE +/- 0.12, N = 3 SE +/- 0.31, N = 4 25.77 25.98 25.62
Apache Spark Row Count: 20000000 - Partitions: 500 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.25, N = 3 SE +/- 0.53, N = 4 148.22 149.14 148.94
Apache Spark Row Count: 20000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.02, N = 3 SE +/- 0.03, N = 4 8.60 8.60 8.57
Apache Spark Row Count: 20000000 - Partitions: 500 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Group By Test Time A B C 3 6 9 12 15 SE +/- 0.08, N = 3 SE +/- 0.06, N = 4 10.12 10.05 10.04
Apache Spark Row Count: 20000000 - Partitions: 500 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Repartition Test Time A B C 4 8 12 16 20 SE +/- 0.11, N = 3 SE +/- 0.07, N = 4 16.94 17.02 17.49
Apache Spark Row Count: 20000000 - Partitions: 500 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Inner Join Test Time A B C 5 10 15 20 25 SE +/- 0.21, N = 3 SE +/- 0.18, N = 4 20.04 19.77 21.21
Apache Spark Row Count: 20000000 - Partitions: 500 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Broadcast Inner Join Test Time A B C 5 10 15 20 25 SE +/- 0.11, N = 3 SE +/- 0.22, N = 4 20.35 20.05 19.94
Apache Spark Row Count: 40000000 - Partitions: 100 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - SHA-512 Benchmark Time A B C 11 22 33 44 55 SE +/- 0.27, N = 3 SE +/- 0.22, N = 3 49.41 49.28 49.53
Apache Spark Row Count: 40000000 - Partitions: 100 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.58, N = 3 SE +/- 0.80, N = 3 149.23 148.60 149.61
Apache Spark Row Count: 40000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.02, N = 3 SE +/- 0.01, N = 3 8.63 8.63 8.65
Apache Spark Row Count: 40000000 - Partitions: 100 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Group By Test Time A B C 6 12 18 24 30 SE +/- 0.13, N = 3 SE +/- 0.17, N = 3 27.36 27.28 27.38
Apache Spark Row Count: 40000000 - Partitions: 100 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Repartition Test Time A B C 8 16 24 32 40 SE +/- 0.38, N = 3 SE +/- 0.20, N = 3 33.93 34.41 34.74
Apache Spark Row Count: 40000000 - Partitions: 100 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Inner Join Test Time A B C 10 20 30 40 50 SE +/- 0.25, N = 3 SE +/- 0.42, N = 3 40.00 39.87 41.98
Apache Spark Row Count: 40000000 - Partitions: 100 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Broadcast Inner Join Test Time A B C 9 18 27 36 45 SE +/- 0.51, N = 3 SE +/- 0.23, N = 3 40.78 40.14 39.62
Apache Spark Row Count: 40000000 - Partitions: 500 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - SHA-512 Benchmark Time A B C 11 22 33 44 55 SE +/- 0.32, N = 3 SE +/- 0.23, N = 3 50.13 49.86 48.84
Apache Spark Row Count: 40000000 - Partitions: 500 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 2.16, N = 3 SE +/- 0.46, N = 3 151.33 149.13 147.72
Apache Spark Row Count: 40000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.03, N = 3 SE +/- 0.05, N = 3 8.59 8.61 8.58
Apache Spark Row Count: 40000000 - Partitions: 500 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Group By Test Time A B C 6 12 18 24 30 SE +/- 0.11, N = 3 SE +/- 0.15, N = 3 23.97 23.49 23.04
Apache Spark Row Count: 40000000 - Partitions: 500 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Repartition Test Time A B C 8 16 24 32 40 SE +/- 0.37, N = 3 SE +/- 0.28, N = 3 33.55 33.85 34.11
Apache Spark Row Count: 40000000 - Partitions: 500 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Inner Join Test Time A B C 9 18 27 36 45 SE +/- 0.60, N = 3 SE +/- 0.14, N = 3 39.55 40.14 40.17
Apache Spark Row Count: 40000000 - Partitions: 500 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Broadcast Inner Join Test Time A B C 9 18 27 36 45 SE +/- 0.39, N = 3 SE +/- 0.53, N = 3 40.32 40.76 40.29
Apache Spark Row Count: 10000000 - Partitions: 1000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - SHA-512 Benchmark Time A B C 4 8 12 16 20 SE +/- 0.09, N = 3 SE +/- 0.10, N = 3 13.97 13.89 14.07
Apache Spark Row Count: 10000000 - Partitions: 1000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.48, N = 3 SE +/- 0.82, N = 3 148.55 148.53 148.99
Apache Spark Row Count: 10000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.03, N = 3 SE +/- 0.01, N = 3 8.49 8.54 8.61
Apache Spark Row Count: 10000000 - Partitions: 1000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Group By Test Time A B C 2 4 6 8 10 SE +/- 0.05, N = 3 SE +/- 0.09, N = 3 6.89 6.77 6.86
Apache Spark Row Count: 10000000 - Partitions: 1000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Repartition Test Time A B C 3 6 9 12 15 SE +/- 0.08, N = 3 SE +/- 0.12, N = 3 9.05 9.01 8.90
Apache Spark Row Count: 10000000 - Partitions: 1000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Inner Join Test Time A B C 3 6 9 12 15 SE +/- 0.41, N = 3 SE +/- 0.17, N = 3 10.79 11.02 10.51
Apache Spark Row Count: 10000000 - Partitions: 1000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Broadcast Inner Join Test Time A B C 3 6 9 12 15 SE +/- 0.43, N = 3 SE +/- 0.41, N = 3 10.65 10.60 10.08
Apache Spark Row Count: 10000000 - Partitions: 2000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - SHA-512 Benchmark Time A B C 4 8 12 16 20 SE +/- 0.06, N = 3 SE +/- 0.10, N = 3 14.15 14.12 14.14
Apache Spark Row Count: 10000000 - Partitions: 2000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.22, N = 3 SE +/- 0.57, N = 3 148.55 149.18 150.28
Apache Spark Row Count: 10000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.032542604, N = 3 SE +/- 0.034828757, N = 3 8.580000000 8.570000000 8.687892725
Apache Spark Row Count: 10000000 - Partitions: 2000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Group By Test Time A B C 2 4 6 8 10 SE +/- 0.08, N = 3 SE +/- 0.02, N = 3 7.03 7.14 7.04
Apache Spark Row Count: 10000000 - Partitions: 2000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Repartition Test Time A B C 3 6 9 12 15 SE +/- 0.07, N = 3 SE +/- 0.03, N = 3 9.34 9.22 9.24
Apache Spark Row Count: 10000000 - Partitions: 2000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Inner Join Test Time A B C 3 6 9 12 15 SE +/- 0.18, N = 3 SE +/- 0.16, N = 3 11.20 11.21 11.66
Apache Spark Row Count: 10000000 - Partitions: 2000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Broadcast Inner Join Test Time A B C 3 6 9 12 15 SE +/- 0.31, N = 3 SE +/- 0.15, N = 3 10.60 10.56 10.41
Apache Spark Row Count: 20000000 - Partitions: 1000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - SHA-512 Benchmark Time A B C 6 12 18 24 30 SE +/- 0.03, N = 3 SE +/- 0.08, N = 3 25.55 25.53 25.60
Apache Spark Row Count: 20000000 - Partitions: 1000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.51, N = 3 SE +/- 0.38, N = 3 149.23 147.91 149.29
Apache Spark Row Count: 20000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.01, N = 3 SE +/- 0.03, N = 3 8.62 8.58 8.54
Apache Spark Row Count: 20000000 - Partitions: 1000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Group By Test Time A B C 3 6 9 12 15 SE +/- 0.09, N = 3 SE +/- 0.04, N = 3 10.25 10.16 10.03
Apache Spark Row Count: 20000000 - Partitions: 1000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Repartition Test Time A B C 4 8 12 16 20 SE +/- 0.08, N = 3 SE +/- 0.04, N = 3 17.03 17.09 17.01
Apache Spark Row Count: 20000000 - Partitions: 1000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Inner Join Test Time A B C 5 10 15 20 25 SE +/- 0.19, N = 3 SE +/- 0.10, N = 3 19.97 20.33 20.49
Apache Spark Row Count: 20000000 - Partitions: 1000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Broadcast Inner Join Test Time A B C 5 10 15 20 25 SE +/- 0.23, N = 3 SE +/- 0.37, N = 3 19.87 20.66 21.08
Apache Spark Row Count: 20000000 - Partitions: 2000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - SHA-512 Benchmark Time A B C 6 12 18 24 30 SE +/- 0.02, N = 3 SE +/- 0.02, N = 3 25.70 25.76 25.72
Apache Spark Row Count: 20000000 - Partitions: 2000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.53, N = 3 SE +/- 0.23, N = 3 148.85 148.30 149.85
Apache Spark Row Count: 20000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.02, N = 3 SE +/- 0.04, N = 3 8.64 8.60 8.59
Apache Spark Row Count: 20000000 - Partitions: 2000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Group By Test Time A B C 3 6 9 12 15 SE +/- 0.08, N = 3 SE +/- 0.04, N = 3 10.25 10.32 10.41
Apache Spark Row Count: 20000000 - Partitions: 2000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Repartition Test Time A B C 4 8 12 16 20 SE +/- 0.23, N = 3 SE +/- 0.29, N = 3 17.52 17.81 17.44
Apache Spark Row Count: 20000000 - Partitions: 2000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Inner Join Test Time A B C 5 10 15 20 25 SE +/- 0.36, N = 3 SE +/- 0.17, N = 3 20.79 20.72 20.50
Apache Spark Row Count: 20000000 - Partitions: 2000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Broadcast Inner Join Test Time A B C 5 10 15 20 25 SE +/- 0.31, N = 3 SE +/- 0.18, N = 3 20.01 19.87 19.31
Apache Spark Row Count: 40000000 - Partitions: 1000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - SHA-512 Benchmark Time A B C 11 22 33 44 55 SE +/- 0.04, N = 3 SE +/- 0.28, N = 3 49.63 49.95 49.84
Apache Spark Row Count: 40000000 - Partitions: 1000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.40, N = 3 SE +/- 0.29, N = 3 149.45 148.66 151.94
Apache Spark Row Count: 40000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.02, N = 3 SE +/- 0.03, N = 3 8.61 8.62 8.56
Apache Spark Row Count: 40000000 - Partitions: 1000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Group By Test Time A B C 6 12 18 24 30 SE +/- 0.20, N = 3 SE +/- 0.23, N = 3 22.89 23.24 23.18
Apache Spark Row Count: 40000000 - Partitions: 1000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Repartition Test Time A B C 8 16 24 32 40 SE +/- 0.28, N = 3 SE +/- 0.43, N = 3 34.50 34.09 34.62
Apache Spark Row Count: 40000000 - Partitions: 1000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Inner Join Test Time A B C 9 18 27 36 45 SE +/- 0.57, N = 3 SE +/- 0.28, N = 3 39.60 40.50 39.90
Apache Spark Row Count: 40000000 - Partitions: 1000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Broadcast Inner Join Test Time A B C 9 18 27 36 45 SE +/- 0.09, N = 3 SE +/- 1.12, N = 3 40.99 41.01 41.20
Apache Spark Row Count: 40000000 - Partitions: 2000 - SHA-512 Benchmark Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - SHA-512 Benchmark Time A B C 12 24 36 48 60 SE +/- 0.17, N = 3 SE +/- 0.68, N = 3 50.33 51.35 50.12
Apache Spark Row Count: 40000000 - Partitions: 2000 - Calculate Pi Benchmark OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Calculate Pi Benchmark A B C 30 60 90 120 150 SE +/- 0.43, N = 3 SE +/- 0.45, N = 3 148.60 148.50 147.98
Apache Spark Row Count: 40000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe A B C 2 4 6 8 10 SE +/- 0.03, N = 3 SE +/- 0.04, N = 3 8.59 8.57 8.50
Apache Spark Row Count: 40000000 - Partitions: 2000 - Group By Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Group By Test Time A B C 6 12 18 24 30 SE +/- 0.21, N = 3 SE +/- 0.07, N = 3 23.26 22.94 23.17
Apache Spark Row Count: 40000000 - Partitions: 2000 - Repartition Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Repartition Test Time A B C 8 16 24 32 40 SE +/- 0.21, N = 3 SE +/- 0.42, N = 3 34.56 33.98 33.96
Apache Spark Row Count: 40000000 - Partitions: 2000 - Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Inner Join Test Time A B C 9 18 27 36 45 SE +/- 0.65, N = 3 SE +/- 0.48, N = 3 40.60 39.58 39.85
Apache Spark Row Count: 40000000 - Partitions: 2000 - Broadcast Inner Join Test Time OpenBenchmarking.org Seconds, Fewer Is Better Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Broadcast Inner Join Test Time A B C 9 18 27 36 45 SE +/- 0.68, N = 3 SE +/- 0.38, N = 3 41.55 40.71 41.18
Phoronix Test Suite v10.8.5