2020-06-24-2159 no tuning or dedup, compression=on, ashift=9 zfs_6x_ssd_in_2x_raidz1: Processor: 2 x Intel Xeon E5-2640 v3 @ 3.40GHz (16 Cores / 32 Threads), Motherboard: Dell 0CNCJW (2.11.0 BIOS), Chipset: Intel Xeon E7 v3/Xeon, Memory: 16 x GB DDR4-2133MT/s HMA42GR7MFR4N-TF, Disk: 6 x 481GB HSCP0480S5xnNMRI + 99GB PERC H730 Mini + 16GB Cruzer Blade + 16GB Cruzer Fit, Graphics: Matrox G200eR2, Monitor: DELL E196FP, Network: 2 x Intel 10-Gigabit X540-AT2 + 2 x Intel I350 + Mellanox MT27500 OS: CentOS Linux 8, Kernel: 4.18.0-193.6.3.el8_2.x86_64 (x86_64), Compiler: GCC 8.3.1 20191121, File-System: xfs, Screen Resolution: 1280x1024 zfs_6x_ssd_in_2x_raidz1_ashift=12: Processor: 2 x Intel Xeon E5-2640 v3 @ 3.40GHz (16 Cores / 32 Threads), Motherboard: Dell 0CNCJW (2.11.0 BIOS), Chipset: Intel Xeon E7 v3/Xeon, Memory: 16 x GB DDR4-2133MT/s HMA42GR7MFR4N-TF, Disk: 6 x 481GB HSCP0480S5xnNMRI + 99GB PERC H730 Mini + 16GB Cruzer Blade + 16GB Cruzer Fit, Graphics: Matrox G200eR2, Monitor: DELL E196FP, Network: 2 x Intel 10-Gigabit X540-AT2 + 2 x Intel I350 + Mellanox MT27500 OS: CentOS Linux 8, Kernel: 4.18.0-193.6.3.el8_2.x86_64 (x86_64), Compiler: GCC 8.3.1 20191121, File-System: xfs, Screen Resolution: 1280x1024 6 x 481GB HSCP0480S5xnNMRI: Processor: 2 x Intel Xeon E5-2640 v3 @ 3.40GHz (16 Cores / 32 Threads), Motherboard: Dell 0CNCJW (2.11.0 BIOS), Chipset: Intel Xeon E7 v3/Xeon, Memory: 16 x GB DDR4-2133MT/s HMA42GR7MFR4N-TF, Disk: 6 x 481GB HSCP0480S5xnNMRI + 99GB PERC H730 Mini + 16GB Cruzer Blade + 16GB Cruzer Fit, Graphics: Matrox G200eR2, Monitor: DELL E196FP, Network: 2 x Intel 10-Gigabit X540-AT2 + 2 x Intel I350 + Mellanox MT27500 OS: CentOS Linux 8, Kernel: 4.18.0-193.6.3.el8_2.x86_64 (x86_64), Compiler: GCC 8.3.1 20191121, File-System: xfs, Screen Resolution: 1280x1024 zfs_ashift=9_compression=on_dedup=off: Processor: 2 x Intel Xeon E5-2640 v3 @ 3.40GHz (16 Cores / 32 Threads), Motherboard: Dell 0CNCJW (2.11.0 BIOS), Chipset: Intel Xeon E7 v3/Xeon, Memory: 16 x GB DDR4-2133MT/s HMA42GR7MFR4N-TF, Disk: 6 x 481GB HSCP0480S5xnNMRI + 99GB PERC H730 Mini + 16GB Cruzer Blade + 16GB Cruzer Fit, Graphics: Matrox G200eR2, Monitor: DELL E196FP, Network: 2 x Intel 10-Gigabit X540-AT2 + 2 x Intel I350 + Mellanox MT27500 OS: CentOS Linux 8, Kernel: 4.18.0-193.6.3.el8_2.x86_64 (x86_64), Compiler: GCC 8.3.1 20191121, File-System: xfs, Screen Resolution: 1280x1024 Flexible IO Tester 3.18 Type: Random Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 946 |================================= 6 x 481GB HSCP0480S5xnNMRI ............ 928 |================================= zfs_ashift=9_compression=on_dedup=off . 968 |================================== Flexible IO Tester 3.18 Type: Random Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 242133 |============================== 6 x 481GB HSCP0480S5xnNMRI ............ 237467 |============================== zfs_ashift=9_compression=on_dedup=off . 247667 |=============================== Flexible IO Tester 3.18 Type: Random Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 2535 |================================= 6 x 481GB HSCP0480S5xnNMRI ............ 2407 |=============================== zfs_ashift=9_compression=on_dedup=off . 2511 |================================= Flexible IO Tester 3.18 Type: Random Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 162267 |=============================== 6 x 481GB HSCP0480S5xnNMRI ............ 154333 |============================= zfs_ashift=9_compression=on_dedup=off . 160667 |=============================== Flexible IO Tester 3.18 Type: Random Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 4553 |================================= 6 x 481GB HSCP0480S5xnNMRI ............ 4260 |=============================== zfs_ashift=9_compression=on_dedup=off . 4394 |================================ Flexible IO Tester 3.18 Type: Random Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 72773 |================================ 6 x 481GB HSCP0480S5xnNMRI ............ 68167 |============================== zfs_ashift=9_compression=on_dedup=off . 70300 |=============================== Flexible IO Tester 3.18 Type: Random Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 211 |====================== 6 x 481GB HSCP0480S5xnNMRI ............ 231 |======================== zfs_ashift=9_compression=on_dedup=off . 321 |================================== Flexible IO Tester 3.18 Type: Random Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 54027 |===================== 6 x 481GB HSCP0480S5xnNMRI ............ 58887 |======================= zfs_ashift=9_compression=on_dedup=off . 82367 |================================ Flexible IO Tester 3.18 Type: Random Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 718 |========================= 6 x 481GB HSCP0480S5xnNMRI ............ 776 |=========================== zfs_ashift=9_compression=on_dedup=off . 961 |================================== Flexible IO Tester 3.18 Type: Random Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 45667 |======================== 6 x 481GB HSCP0480S5xnNMRI ............ 49520 |========================== zfs_ashift=9_compression=on_dedup=off . 61533 |================================ Flexible IO Tester 3.18 Type: Random Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 1879 |================================= 6 x 481GB HSCP0480S5xnNMRI ............ 1865 |================================= zfs_ashift=9_compression=on_dedup=off . 1669 |============================= Flexible IO Tester 3.18 Type: Random Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 30067 |================================ 6 x 481GB HSCP0480S5xnNMRI ............ 29867 |================================ zfs_ashift=9_compression=on_dedup=off . 26700 |============================ Flexible IO Tester 3.18 Type: Sequential Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 1396 |================================ 6 x 481GB HSCP0480S5xnNMRI ............ 1416 |================================ zfs_ashift=9_compression=on_dedup=off . 1447 |================================= Flexible IO Tester 3.18 Type: Sequential Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 357667 |============================== 6 x 481GB HSCP0480S5xnNMRI ............ 362667 |============================== zfs_ashift=9_compression=on_dedup=off . 371000 |=============================== Flexible IO Tester 3.18 Type: Random Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 872 |================================== 6 x 481GB HSCP0480S5xnNMRI ............ 860 |================================== zfs_ashift=9_compression=on_dedup=off . 868 |================================== Flexible IO Tester 3.18 Type: Random Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 223333 |=============================== 6 x 481GB HSCP0480S5xnNMRI ............ 220000 |=============================== zfs_ashift=9_compression=on_dedup=off . 222333 |=============================== Flexible IO Tester 3.18 Type: Sequential Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 3254 |================================= 6 x 481GB HSCP0480S5xnNMRI ............ 3244 |================================= zfs_ashift=9_compression=on_dedup=off . 3233 |================================= Flexible IO Tester 3.18 Type: Sequential Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 208333 |=============================== 6 x 481GB HSCP0480S5xnNMRI ............ 207667 |=============================== zfs_ashift=9_compression=on_dedup=off . 207000 |=============================== Flexible IO Tester 3.18 Type: Sequential Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 4822 |================================ 6 x 481GB HSCP0480S5xnNMRI ............ 4897 |================================= zfs_ashift=9_compression=on_dedup=off . 4809 |================================ Flexible IO Tester 3.18 Type: Sequential Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 77167 |================================ 6 x 481GB HSCP0480S5xnNMRI ............ 78333 |================================ zfs_ashift=9_compression=on_dedup=off . 76633 |=============================== Flexible IO Tester 3.18 Type: Sequential Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 700 |================================ 6 x 481GB HSCP0480S5xnNMRI ............ 703 |================================ zfs_ashift=9_compression=on_dedup=off . 743 |================================== Flexible IO Tester 3.18 Type: Sequential Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 179333 |============================= 6 x 481GB HSCP0480S5xnNMRI ............ 180333 |============================= zfs_ashift=9_compression=on_dedup=off . 190333 |=============================== Flexible IO Tester 3.18 Type: Random Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 2288 |================================= 6 x 481GB HSCP0480S5xnNMRI ............ 2283 |================================ zfs_ashift=9_compression=on_dedup=off . 2321 |================================= Flexible IO Tester 3.18 Type: Random Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 146667 |=============================== 6 x 481GB HSCP0480S5xnNMRI ............ 146000 |============================== zfs_ashift=9_compression=on_dedup=off . 148667 |=============================== Flexible IO Tester 3.18 Type: Random Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 4092 |================================= 6 x 481GB HSCP0480S5xnNMRI ............ 4153 |================================= zfs_ashift=9_compression=on_dedup=off . 4117 |================================= Flexible IO Tester 3.18 Type: Random Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 65450 |================================ 6 x 481GB HSCP0480S5xnNMRI ............ 66256 |================================ zfs_ashift=9_compression=on_dedup=off . 65867 |================================ Flexible IO Tester 3.18 Type: Random Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 212 |================================= 6 x 481GB HSCP0480S5xnNMRI ............ 218 |================================== zfs_ashift=9_compression=on_dedup=off . 181 |============================ Flexible IO Tester 3.18 Type: Random Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 54133 |=============================== 6 x 481GB HSCP0480S5xnNMRI ............ 55667 |================================ zfs_ashift=9_compression=on_dedup=off . 46300 |=========================== Flexible IO Tester 3.18 Type: Sequential Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 1625 |============================= 6 x 481GB HSCP0480S5xnNMRI ............ 1581 |============================ zfs_ashift=9_compression=on_dedup=off . 1862 |================================= Flexible IO Tester 3.18 Type: Sequential Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 104000 |=========================== 6 x 481GB HSCP0480S5xnNMRI ............ 101150 |========================== zfs_ashift=9_compression=on_dedup=off . 119000 |=============================== Flexible IO Tester 3.18 Type: Sequential Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 2106 |================================ 6 x 481GB HSCP0480S5xnNMRI ............ 2097 |================================ zfs_ashift=9_compression=on_dedup=off . 2176 |================================= Flexible IO Tester 3.18 Type: Sequential Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 33475 |=============================== 6 x 481GB HSCP0480S5xnNMRI ............ 33567 |=============================== zfs_ashift=9_compression=on_dedup=off . 34800 |================================ Flexible IO Tester 3.18 Type: Random Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 662 |=============================== 6 x 481GB HSCP0480S5xnNMRI ............ 726 |================================== zfs_ashift=9_compression=on_dedup=off . 620 |============================= Flexible IO Tester 3.18 Type: Random Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 42225 |============================= 6 x 481GB HSCP0480S5xnNMRI ............ 46364 |================================ zfs_ashift=9_compression=on_dedup=off . 39640 |=========================== Flexible IO Tester 3.18 Type: Random Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 1611 |================================= 6 x 481GB HSCP0480S5xnNMRI ............ 1613 |================================= zfs_ashift=9_compression=on_dedup=off . 1544 |================================ Flexible IO Tester 3.18 Type: Random Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 25707 |================================ 6 x 481GB HSCP0480S5xnNMRI ............ 25800 |================================ zfs_ashift=9_compression=on_dedup=off . 24700 |=============================== Flexible IO Tester 3.18 Type: Sequential Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 1086 |================================ 6 x 481GB HSCP0480S5xnNMRI ............ 1117 |================================= zfs_ashift=9_compression=on_dedup=off . 1131 |================================= Flexible IO Tester 3.18 Type: Sequential Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 278333 |============================== 6 x 481GB HSCP0480S5xnNMRI ............ 286333 |=============================== zfs_ashift=9_compression=on_dedup=off . 289667 |=============================== Flexible IO Tester 3.18 Type: Sequential Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 2753 |================================ 6 x 481GB HSCP0480S5xnNMRI ............ 2820 |================================= zfs_ashift=9_compression=on_dedup=off . 2821 |================================= Flexible IO Tester 3.18 Type: Sequential Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 176000 |============================== 6 x 481GB HSCP0480S5xnNMRI ............ 180667 |=============================== zfs_ashift=9_compression=on_dedup=off . 180600 |=============================== Flexible IO Tester 3.18 Type: Sequential Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 4385 |================================ 6 x 481GB HSCP0480S5xnNMRI ............ 4495 |================================= zfs_ashift=9_compression=on_dedup=off . 4454 |================================= Flexible IO Tester 3.18 Type: Sequential Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 70167 |=============================== 6 x 481GB HSCP0480S5xnNMRI ............ 71900 |================================ zfs_ashift=9_compression=on_dedup=off . 71260 |================================ Flexible IO Tester 3.18 Type: Sequential Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 615 |================================= 6 x 481GB HSCP0480S5xnNMRI ............ 628 |================================== zfs_ashift=9_compression=on_dedup=off . 614 |================================= Flexible IO Tester 3.18 Type: Sequential Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 157333 |============================== 6 x 481GB HSCP0480S5xnNMRI ............ 161000 |=============================== zfs_ashift=9_compression=on_dedup=off . 157000 |============================== Flexible IO Tester 3.18 Type: Sequential Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 1423 |================================= 6 x 481GB HSCP0480S5xnNMRI ............ 1441 |================================= zfs_ashift=9_compression=on_dedup=off . 1441 |================================= Flexible IO Tester 3.18 Type: Sequential Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 91067 |================================ 6 x 481GB HSCP0480S5xnNMRI ............ 92233 |================================ zfs_ashift=9_compression=on_dedup=off . 92187 |================================ Flexible IO Tester 3.18 Type: Sequential Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1 MB/s > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 1811 |============================= 6 x 481GB HSCP0480S5xnNMRI ............ 2072 |================================= zfs_ashift=9_compression=on_dedup=off . 1961 |=============================== Flexible IO Tester 3.18 Type: Sequential Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1 IOPS > Higher Is Better zfs_6x_ssd_in_2x_raidz1 ............... 28967 |============================ 6 x 481GB HSCP0480S5xnNMRI ............ 33167 |================================ zfs_ashift=9_compression=on_dedup=off . 31600 |==============================