spark rembrandty AMD Ryzen 7 PRO 6850U testing with a LENOVO 21CM0001US (R22ET46W 1.16 BIOS) and AMD Rembrandt 1GB on Ubuntu 22.04 via the Phoronix Test Suite. A: Processor: AMD Ryzen 7 PRO 6850U @ 4.77GHz (8 Cores / 16 Threads), Motherboard: LENOVO 21CM0001US (R22ET46W 1.16 BIOS), Chipset: AMD Device 14b5, Memory: 16GB, Disk: 512GB Micron MTFDKBA512TFK, Graphics: AMD Rembrandt 1GB, Audio: AMD Rembrandt Radeon HD Audio, Network: Qualcomm Atheros QCNFA765 OS: Ubuntu 22.04, Kernel: 5.15.0-41-generic (x86_64), Desktop: GNOME Shell 42.2, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 22.2.0-devel (git-ba461f8 2022-07-26 jammy-oibaf-ppa) (LLVM 14.0.6 DRM 3.42), Vulkan: 1.3.219, Compiler: GCC 11.2.0, File-System: ext4, Screen Resolution: 1920x1200 B: Processor: AMD Ryzen 7 PRO 6850U @ 4.77GHz (8 Cores / 16 Threads), Motherboard: LENOVO 21CM0001US (R22ET46W 1.16 BIOS), Chipset: AMD Device 14b5, Memory: 16GB, Disk: 512GB Micron MTFDKBA512TFK, Graphics: AMD Rembrandt 1GB, Audio: AMD Rembrandt Radeon HD Audio, Network: Qualcomm Atheros QCNFA765 OS: Ubuntu 22.04, Kernel: 5.15.0-41-generic (x86_64), Desktop: GNOME Shell 42.2, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 22.2.0-devel (git-ba461f8 2022-07-26 jammy-oibaf-ppa) (LLVM 14.0.6 DRM 3.42), Vulkan: 1.3.219, Compiler: GCC 11.2.0, File-System: ext4, Screen Resolution: 1920x1200 Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - SHA-512 Benchmark Time Seconds < Lower Is Better A . 4.54 |==================================================================== B . 4.60 |===================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark Seconds < Lower Is Better A . 241.05 |=================================================================== B . 241.02 |=================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe Seconds < Lower Is Better A . 13.62 |==================================================================== B . 13.49 |=================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Group By Test Time Seconds < Lower Is Better A . 4.57 |===================================================================== B . 4.48 |==================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Repartition Test Time Seconds < Lower Is Better A . 3.37 |===================================================================== B . 3.24 |================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Inner Join Test Time Seconds < Lower Is Better A . 2.44 |===================================================================== B . 2.39 |==================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 100 - Broadcast Inner Join Test Time Seconds < Lower Is Better A . 2.14 |===================================================================== B . 1.89 |============================================================= Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - SHA-512 Benchmark Time Seconds < Lower Is Better A . 4.75 |===================================================================== B . 4.74 |===================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Calculate Pi Benchmark Seconds < Lower Is Better A . 240.04 |================================================================== B . 245.15 |=================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe Seconds < Lower Is Better A . 13.66 |==================================================================== B . 13.44 |=================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Group By Test Time Seconds < Lower Is Better A . 4.92 |=================================================================== B . 5.03 |===================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Repartition Test Time Seconds < Lower Is Better A . 3.40 |==================================================================== B . 3.43 |===================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Inner Join Test Time Seconds < Lower Is Better A . 2.65 |===================================================================== B . 2.53 |================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 500 - Broadcast Inner Join Test Time Seconds < Lower Is Better A . 3.08 |===================================================================== B . 2.29 |=================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - SHA-512 Benchmark Time Seconds < Lower Is Better A . 5.11 |===================================================================== B . 5.08 |===================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Calculate Pi Benchmark Seconds < Lower Is Better A . 239.97 |=================================================================== B . 239.65 |=================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe Seconds < Lower Is Better A . 13.50 |==================================================================== B . 13.46 |==================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Group By Test Time Seconds < Lower Is Better A . 5.32 |================================================================== B . 5.55 |===================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Repartition Test Time Seconds < Lower Is Better A . 3.68 |===================================================================== B . 3.70 |===================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Inner Join Test Time Seconds < Lower Is Better A . 3.17 |===================================================================== B . 3.17 |===================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 1000 - Broadcast Inner Join Test Time Seconds < Lower Is Better A . 3.21 |===================================================================== B . 2.62 |======================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - SHA-512 Benchmark Time Seconds < Lower Is Better A . 5.63 |===================================================================== B . 5.46 |=================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark Seconds < Lower Is Better A . 241.47 |=================================================================== B . 242.20 |=================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe Seconds < Lower Is Better A . 13.59 |==================================================================== B . 13.54 |==================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Group By Test Time Seconds < Lower Is Better A . 6.09 |===================================================================== B . 6.10 |===================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Repartition Test Time Seconds < Lower Is Better A . 4.15 |===================================================================== B . 4.09 |==================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Inner Join Test Time Seconds < Lower Is Better A . 4.01 |===================================================================== B . 3.93 |==================================================================== Apache Spark 3.3 Row Count: 1000000 - Partitions: 2000 - Broadcast Inner Join Test Time Seconds < Lower Is Better A . 3.31 |===================================================================== B . 3.04 |=============================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - SHA-512 Benchmark Time Seconds < Lower Is Better A . 19.58 |==================================================================== B . 19.62 |==================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Calculate Pi Benchmark Seconds < Lower Is Better A . 238.65 |=================================================================== B . 240.01 |=================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe Seconds < Lower Is Better A . 13.60 |==================================================================== B . 13.54 |==================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Group By Test Time Seconds < Lower Is Better A . 10.00 |==================================================================== B . 9.72 |================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Repartition Test Time Seconds < Lower Is Better A . 14.45 |=================================================================== B . 14.63 |==================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Inner Join Test Time Seconds < Lower Is Better A . 16.71 |==================================================================== B . 15.76 |================================================================ Apache Spark 3.3 Row Count: 10000000 - Partitions: 100 - Broadcast Inner Join Test Time Seconds < Lower Is Better A . 16.22 |==================================================================== B . 15.13 |=============================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - SHA-512 Benchmark Time Seconds < Lower Is Better A . 19.70 |==================================================================== B . 19.79 |==================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Calculate Pi Benchmark Seconds < Lower Is Better A . 239.05 |=================================================================== B . 240.25 |=================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe Seconds < Lower Is Better A . 13.56 |==================================================================== B . 13.47 |==================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Group By Test Time Seconds < Lower Is Better A . 9.86 |=================================================================== B . 10.02 |==================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Repartition Test Time Seconds < Lower Is Better A . 14.07 |==================================================================== B . 13.94 |=================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Inner Join Test Time Seconds < Lower Is Better A . 15.11 |=================================================================== B . 15.41 |==================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 500 - Broadcast Inner Join Test Time Seconds < Lower Is Better A . 15.05 |=================================================================== B . 15.17 |==================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - SHA-512 Benchmark Time Seconds < Lower Is Better A . 36.52 |==================================================================== B . 36.48 |==================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Calculate Pi Benchmark Seconds < Lower Is Better A . 239.84 |=================================================================== B . 240.03 |=================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe Seconds < Lower Is Better A . 13.51 |=================================================================== B . 13.67 |==================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Group By Test Time Seconds < Lower Is Better A . 15.37 |==================================================================== B . 14.90 |================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Repartition Test Time Seconds < Lower Is Better A . 27.45 |==================================================================== B . 27.02 |=================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Inner Join Test Time Seconds < Lower Is Better A . 32.48 |==================================================================== B . 31.13 |================================================================= Apache Spark 3.3 Row Count: 20000000 - Partitions: 100 - Broadcast Inner Join Test Time Seconds < Lower Is Better A . 32.10 |==================================================================== B . 31.82 |=================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - SHA-512 Benchmark Time Seconds < Lower Is Better A . 35.00 |==================================================================== B . 34.98 |==================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Calculate Pi Benchmark Seconds < Lower Is Better A . 241.40 |=================================================================== B . 239.40 |================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe Seconds < Lower Is Better A . 13.50 |==================================================================== B . 13.45 |==================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Group By Test Time Seconds < Lower Is Better A . 14.92 |==================================================================== B . 14.84 |==================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Repartition Test Time Seconds < Lower Is Better A . 26.27 |==================================================================== B . 25.55 |================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Inner Join Test Time Seconds < Lower Is Better A . 30.27 |==================================================================== B . 29.12 |================================================================= Apache Spark 3.3 Row Count: 20000000 - Partitions: 500 - Broadcast Inner Join Test Time Seconds < Lower Is Better A . 28.64 |==================================================================== B . 28.30 |=================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - SHA-512 Benchmark Time Seconds < Lower Is Better A . 69.06 |==================================================================== B . 66.03 |================================================================= Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Calculate Pi Benchmark Seconds < Lower Is Better A . 249.74 |=================================================================== B . 240.99 |================================================================= Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Calculate Pi Benchmark Using Dataframe Seconds < Lower Is Better A . 13.83 |==================================================================== B . 13.70 |=================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Group By Test Time Seconds < Lower Is Better A . 40.60 |==================================================================== B . 40.43 |==================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Repartition Test Time Seconds < Lower Is Better A . 52.39 |==================================================================== B . 49.82 |================================================================= Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Inner Join Test Time Seconds < Lower Is Better A . 61.59 |==================================================================== B . 56.68 |=============================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 100 - Broadcast Inner Join Test Time Seconds < Lower Is Better A . 60.30 |==================================================================== B . 58.25 |================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - SHA-512 Benchmark Time Seconds < Lower Is Better A . 69.51 |==================================================================== B . 68.08 |=================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Calculate Pi Benchmark Seconds < Lower Is Better A . 249.44 |=================================================================== B . 244.74 |================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Calculate Pi Benchmark Using Dataframe Seconds < Lower Is Better A . 13.66 |==================================================================== B . 13.43 |=================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Group By Test Time Seconds < Lower Is Better A . 35.35 |=================================================================== B . 35.64 |==================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Repartition Test Time Seconds < Lower Is Better A . 50.68 |==================================================================== B . 49.20 |================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Inner Join Test Time Seconds < Lower Is Better A . 61.35 |==================================================================== B . 57.43 |================================================================ Apache Spark 3.3 Row Count: 40000000 - Partitions: 500 - Broadcast Inner Join Test Time Seconds < Lower Is Better A . 64.33 |==================================================================== B . 58.92 |============================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - SHA-512 Benchmark Time Seconds < Lower Is Better A . 19.99 |==================================================================== B . 20.05 |==================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Calculate Pi Benchmark Seconds < Lower Is Better A . 241.82 |=================================================================== B . 239.84 |================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe Seconds < Lower Is Better A . 13.56 |==================================================================== B . 13.58 |==================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Group By Test Time Seconds < Lower Is Better A . 10.44 |=================================================================== B . 10.65 |==================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Repartition Test Time Seconds < Lower Is Better A . 13.96 |=================================================================== B . 14.12 |==================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Inner Join Test Time Seconds < Lower Is Better A . 15.58 |==================================================================== B . 15.67 |==================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 1000 - Broadcast Inner Join Test Time Seconds < Lower Is Better A . 14.92 |==================================================================== B . 15.03 |==================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - SHA-512 Benchmark Time Seconds < Lower Is Better A . 20.40 |==================================================================== B . 20.11 |=================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Calculate Pi Benchmark Seconds < Lower Is Better A . 241.94 |=================================================================== B . 241.11 |=================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe Seconds < Lower Is Better A . 13.54 |==================================================================== B . 13.60 |==================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Group By Test Time Seconds < Lower Is Better A . 10.94 |==================================================================== B . 10.50 |================================================================= Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Repartition Test Time Seconds < Lower Is Better A . 14.44 |==================================================================== B . 14.17 |=================================================================== Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Inner Join Test Time Seconds < Lower Is Better A . 16.80 |==================================================================== B . 16.17 |================================================================= Apache Spark 3.3 Row Count: 10000000 - Partitions: 2000 - Broadcast Inner Join Test Time Seconds < Lower Is Better A . 15.63 |==================================================================== B . 15.34 |=================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - SHA-512 Benchmark Time Seconds < Lower Is Better A . 35.73 |=================================================================== B . 36.03 |==================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Calculate Pi Benchmark Seconds < Lower Is Better A . 241.00 |================================================================== B . 244.01 |=================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe Seconds < Lower Is Better A . 13.60 |==================================================================== B . 13.53 |==================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Group By Test Time Seconds < Lower Is Better A . 14.95 |================================================================= B . 15.58 |==================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Repartition Test Time Seconds < Lower Is Better A . 25.65 |================================================================= B . 26.99 |==================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Inner Join Test Time Seconds < Lower Is Better A . 30.15 |=================================================================== B . 30.55 |==================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 1000 - Broadcast Inner Join Test Time Seconds < Lower Is Better A . 28.77 |================================================================= B . 30.29 |==================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - SHA-512 Benchmark Time Seconds < Lower Is Better A . 36.28 |==================================================================== B . 36.36 |==================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Calculate Pi Benchmark Seconds < Lower Is Better A . 242.09 |=================================================================== B . 243.77 |=================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe Seconds < Lower Is Better A . 13.58 |==================================================================== B . 13.48 |=================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Group By Test Time Seconds < Lower Is Better A . 15.78 |==================================================================== B . 15.79 |==================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Repartition Test Time Seconds < Lower Is Better A . 26.46 |==================================================================== B . 26.54 |==================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Inner Join Test Time Seconds < Lower Is Better A . 30.36 |=================================================================== B . 30.83 |==================================================================== Apache Spark 3.3 Row Count: 20000000 - Partitions: 2000 - Broadcast Inner Join Test Time Seconds < Lower Is Better A . 29.44 |==================================================================== B . 29.56 |==================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - SHA-512 Benchmark Time Seconds < Lower Is Better A . 69.98 |==================================================================== B . 69.51 |==================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Calculate Pi Benchmark Seconds < Lower Is Better A . 247.40 |=================================================================== B . 246.28 |=================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Calculate Pi Benchmark Using Dataframe Seconds < Lower Is Better A . 13.58 |==================================================================== B . 13.65 |==================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Group By Test Time Seconds < Lower Is Better A . 33.95 |================================================================== B . 34.84 |==================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Repartition Test Time Seconds < Lower Is Better A . 50.05 |==================================================================== B . 50.27 |==================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Inner Join Test Time Seconds < Lower Is Better A . 59.25 |=================================================================== B . 59.74 |==================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 1000 - Broadcast Inner Join Test Time Seconds < Lower Is Better A . 59.41 |==================================================================== B . 59.06 |==================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - SHA-512 Benchmark Time Seconds < Lower Is Better A . 70.44 |==================================================================== B . 70.43 |==================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Calculate Pi Benchmark Seconds < Lower Is Better A . 246.54 |================================================================== B . 250.15 |=================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Calculate Pi Benchmark Using Dataframe Seconds < Lower Is Better A . 13.83 |==================================================================== B . 13.73 |==================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Group By Test Time Seconds < Lower Is Better A . 34.71 |==================================================================== B . 34.91 |==================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Repartition Test Time Seconds < Lower Is Better A . 50.81 |==================================================================== B . 50.24 |=================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Inner Join Test Time Seconds < Lower Is Better A . 61.61 |==================================================================== B . 61.55 |==================================================================== Apache Spark 3.3 Row Count: 40000000 - Partitions: 2000 - Broadcast Inner Join Test Time Seconds < Lower Is Better A . 61.85 |==================================================================== B . 61.79 |====================================================================