pts-disk-different-nvmes AMD Ryzen Threadripper 1900X 8-Core testing with a Gigabyte X399 DESIGNARE EX-CF (F13a BIOS) and NVIDIA Quadro P400 on Debian 11 via the Phoronix Test Suite. ZFS zraid1 4xNVME Pool: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1920x1080 ext4 mdadm raid5 4xNVME: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1920x1080 ext4 Crucial P5 Plus 1TB NVME: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1920x1080 ZFS zraid1 8xNVME Pool: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1920x1080 ZFS zraid1 8xNVME Pool no Compression: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1920x1080 ZFS mirror 8xNVME Pool: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1920x1080 SQLite 3.30.1 Threads / Copies: 1 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool ................ 7.852 |============================= ext4 mdadm raid5 4xNVME ............... 8.810 |================================ ext4 Crucial P5 Plus 1TB NVME ......... 8.367 |============================== ZFS zraid1 8xNVME Pool ................ 8.380 |============================== ZFS zraid1 8xNVME Pool no Compression . 8.379 |============================== ZFS mirror 8xNVME Pool ................ 7.252 |========================== SQLite 3.30.1 Threads / Copies: 8 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool ................ 11.267 |================= ext4 mdadm raid5 4xNVME ............... 20.005 |=============================== ext4 Crucial P5 Plus 1TB NVME ......... 15.767 |======================== ZFS zraid1 8xNVME Pool ................ 12.857 |==================== ZFS zraid1 8xNVME Pool no Compression . 12.827 |==================== ZFS mirror 8xNVME Pool ................ 9.043 |============== SQLite 3.30.1 Threads / Copies: 32 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool ................ 23.19 |======================= ext4 mdadm raid5 4xNVME ............... 31.79 |================================ ext4 Crucial P5 Plus 1TB NVME ......... 27.58 |============================ ZFS zraid1 8xNVME Pool ................ 27.00 |=========================== ZFS zraid1 8xNVME Pool no Compression . 26.89 |=========================== ZFS mirror 8xNVME Pool ................ 17.74 |================== SQLite 3.30.1 Threads / Copies: 64 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool ................ 37.45 |========================== ext4 mdadm raid5 4xNVME ............... 45.72 |================================ ext4 Crucial P5 Plus 1TB NVME ......... 40.98 |============================= ZFS zraid1 8xNVME Pool ................ 40.30 |============================ ZFS zraid1 8xNVME Pool no Compression . 40.00 |============================ ZFS mirror 8xNVME Pool ................ 36.60 |========================== SQLite 3.30.1 Threads / Copies: 128 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool ................ 76.09 |============================== ext4 mdadm raid5 4xNVME ............... 80.15 |================================ ext4 Crucial P5 Plus 1TB NVME ......... 59.89 |======================== ZFS zraid1 8xNVME Pool ................ 78.15 |=============================== ZFS zraid1 8xNVME Pool no Compression . 77.05 |=============================== ZFS mirror 8xNVME Pool ................ 77.88 |=============================== Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 4245 |===================== ext4 Crucial P5 Plus 1TB NVME ......... 3538 |================== ZFS zraid1 8xNVME Pool ................ 4037 |==================== ZFS zraid1 8xNVME Pool no Compression . 4170 |===================== ZFS mirror 8xNVME Pool ................ 6617 |================================= Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ................ 2119 |========== ext4 mdadm raid5 4xNVME ............... 6666 |================================= ext4 Crucial P5 Plus 1TB NVME ......... 1765 |========= ZFS zraid1 8xNVME Pool ................ 2015 |========== ZFS zraid1 8xNVME Pool no Compression . 2081 |========== ZFS mirror 8xNVME Pool ................ 3305 |================ Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 194 |===== ext4 mdadm raid5 4xNVME ............... 1199 |=============================== ext4 Crucial P5 Plus 1TB NVME ......... 1286 |================================= ZFS zraid1 8xNVME Pool ................ 224 |====== ZFS zraid1 8xNVME Pool no Compression . 219 |====== ZFS mirror 8xNVME Pool ................ 877 |======================= Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ................ 49567 |===== ext4 mdadm raid5 4xNVME ............... 306667 |============================= ext4 Crucial P5 Plus 1TB NVME ......... 329667 |=============================== ZFS zraid1 8xNVME Pool ................ 57433 |===== ZFS zraid1 8xNVME Pool no Compression . 56167 |===== ZFS mirror 8xNVME Pool ................ 224333 |===================== Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 2458 |======================= ext4 mdadm raid5 4xNVME ............... 958 |========= ext4 Crucial P5 Plus 1TB NVME ......... 3252 |============================== ZFS zraid1 8xNVME Pool ................ 2661 |========================= ZFS zraid1 8xNVME Pool no Compression . 2681 |========================= ZFS mirror 8xNVME Pool ................ 3573 |================================= Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ................ 1225 |======================= ext4 mdadm raid5 4xNVME ............... 476 |========= ext4 Crucial P5 Plus 1TB NVME ......... 1623 |============================== ZFS zraid1 8xNVME Pool ................ 1327 |========================= ZFS zraid1 8xNVME Pool no Compression . 1337 |========================= ZFS mirror 8xNVME Pool ................ 1783 |================================= Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 162 |====== ext4 mdadm raid5 4xNVME ............... 324 |============= ext4 Crucial P5 Plus 1TB NVME ......... 878 |================================== ZFS zraid1 8xNVME Pool ................ 177 |======= ZFS zraid1 8xNVME Pool no Compression . 188 |======= ZFS mirror 8xNVME Pool ................ 267 |========== Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ................ 41500 |====== ext4 mdadm raid5 4xNVME ............... 82967 |=========== ext4 Crucial P5 Plus 1TB NVME ......... 224667 |=============================== ZFS zraid1 8xNVME Pool ................ 45367 |====== ZFS zraid1 8xNVME Pool no Compression . 48200 |======= ZFS mirror 8xNVME Pool ................ 68333 |========= Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 4075 |================================ ext4 Crucial P5 Plus 1TB NVME ......... 3538 |============================ ZFS zraid1 8xNVME Pool ................ 4005 |================================ ZFS zraid1 8xNVME Pool no Compression . 3998 |================================ ZFS mirror 8xNVME Pool ................ 4154 |================================= Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ................ 2034 |========== ext4 mdadm raid5 4xNVME ............... 6706 |================================= ext4 Crucial P5 Plus 1TB NVME ......... 1765 |========= ZFS zraid1 8xNVME Pool ................ 1999 |========== ZFS zraid1 8xNVME Pool no Compression . 1995 |========== ZFS mirror 8xNVME Pool ................ 2073 |========== Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 1287 |================================ ext4 mdadm raid5 4xNVME ............... 865 |====================== ext4 Crucial P5 Plus 1TB NVME ......... 1311 |================================= ZFS zraid1 8xNVME Pool ................ 1275 |================================ ZFS zraid1 8xNVME Pool no Compression . 1294 |================================= ZFS mirror 8xNVME Pool ................ 1296 |================================= Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ................ 329333 |============================== ext4 mdadm raid5 4xNVME ............... 221333 |==================== ext4 Crucial P5 Plus 1TB NVME ......... 335667 |=============================== ZFS zraid1 8xNVME Pool ................ 326333 |============================== ZFS zraid1 8xNVME Pool no Compression . 331000 |=============================== ZFS mirror 8xNVME Pool ................ 332000 |=============================== Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 2065 |=================== ext4 mdadm raid5 4xNVME ............... 1308 |============ ext4 Crucial P5 Plus 1TB NVME ......... 3236 |============================== ZFS zraid1 8xNVME Pool ................ 2381 |====================== ZFS zraid1 8xNVME Pool no Compression . 2446 |======================= ZFS mirror 8xNVME Pool ................ 3555 |================================= Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ................ 1029 |=================== ext4 mdadm raid5 4xNVME ............... 650 |============ ext4 Crucial P5 Plus 1TB NVME ......... 1614 |============================== ZFS zraid1 8xNVME Pool ................ 1186 |====================== ZFS zraid1 8xNVME Pool no Compression . 1220 |======================= ZFS mirror 8xNVME Pool ................ 1774 |================================= Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 670 |========================== ext4 mdadm raid5 4xNVME ............... 449 |================= ext4 Crucial P5 Plus 1TB NVME ......... 891 |================================== ZFS zraid1 8xNVME Pool ................ 665 |========================= ZFS zraid1 8xNVME Pool no Compression . 682 |========================== ZFS mirror 8xNVME Pool ................ 771 |============================= Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ................ 171333 |======================= ext4 mdadm raid5 4xNVME ............... 115000 |================ ext4 Crucial P5 Plus 1TB NVME ......... 228000 |=============================== ZFS zraid1 8xNVME Pool ................ 170667 |======================= ZFS zraid1 8xNVME Pool no Compression . 174667 |======================== ZFS mirror 8xNVME Pool ................ 197333 |=========================== FS-Mark 3.3 Test: 1000 Files, 1MB Size Files/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 646.5 |================================ ext4 mdadm raid5 4xNVME ............... 254.0 |============= ext4 Crucial P5 Plus 1TB NVME ......... 584.1 |============================= ZFS zraid1 8xNVME Pool ................ 646.7 |================================ ZFS zraid1 8xNVME Pool no Compression . 589.7 |============================= ZFS mirror 8xNVME Pool ................ 581.1 |============================= FS-Mark 3.3 Test: 5000 Files, 1MB Size, 4 Threads Files/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 1632.7 |============================== ext4 mdadm raid5 4xNVME ............... 537.9 |========== ext4 Crucial P5 Plus 1TB NVME ......... 522.1 |========= ZFS zraid1 8xNVME Pool ................ 1655.6 |============================== ZFS zraid1 8xNVME Pool no Compression . 1240.0 |====================== ZFS mirror 8xNVME Pool ................ 1711.4 |=============================== FS-Mark 3.3 Test: 4000 Files, 32 Sub Dirs, 1MB Size Files/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 684.2 |================================ ext4 mdadm raid5 4xNVME ............... 284.7 |============= ext4 Crucial P5 Plus 1TB NVME ......... 435.0 |==================== ZFS zraid1 8xNVME Pool ................ 644.7 |============================== ZFS zraid1 8xNVME Pool no Compression . 592.1 |============================ ZFS mirror 8xNVME Pool ................ 658.2 |=============================== FS-Mark 3.3 Test: 1000 Files, 1MB Size, No Sync/FSync Files/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 1359.3 |======================== ext4 mdadm raid5 4xNVME ............... 1733.3 |============================== ext4 Crucial P5 Plus 1TB NVME ......... 1791.6 |=============================== ZFS zraid1 8xNVME Pool ................ 1340.7 |======================= ZFS zraid1 8xNVME Pool no Compression . 1160.9 |==================== ZFS mirror 8xNVME Pool ................ 1338.4 |======================= Dbench 4.0 12 Clients MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 2626.03 |============================ ext4 mdadm raid5 4xNVME ............... 2484.85 |=========================== ext4 Crucial P5 Plus 1TB NVME ......... 658.79 |======= ZFS zraid1 8xNVME Pool ................ 2472.42 |=========================== ZFS zraid1 8xNVME Pool no Compression . 2783.64 |============================== ZFS mirror 8xNVME Pool ................ 2730.75 |============================= Dbench 4.0 1 Clients MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 410.17 |============================ ext4 mdadm raid5 4xNVME ............... 453.17 |=============================== ext4 Crucial P5 Plus 1TB NVME ......... 102.99 |======= ZFS zraid1 8xNVME Pool ................ 380.35 |========================== ZFS zraid1 8xNVME Pool no Compression . 458.91 |=============================== ZFS mirror 8xNVME Pool ................ 435.70 |============================= Compile Bench 0.6 Test: Compile MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 1398.57 |============================ ext4 mdadm raid5 4xNVME ............... 1478.00 |============================== ext4 Crucial P5 Plus 1TB NVME ......... 1483.07 |============================== ZFS zraid1 8xNVME Pool ................ 1390.71 |============================ ZFS zraid1 8xNVME Pool no Compression . 1247.36 |========================= ZFS mirror 8xNVME Pool ................ 1408.42 |============================ Compile Bench 0.6 Test: Initial Create MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 222.04 |================ ext4 mdadm raid5 4xNVME ............... 411.07 |============================== ext4 Crucial P5 Plus 1TB NVME ......... 423.67 |=============================== ZFS zraid1 8xNVME Pool ................ 221.83 |================ ZFS zraid1 8xNVME Pool no Compression . 206.77 |=============== ZFS mirror 8xNVME Pool ................ 216.21 |================ Compile Bench 0.6 Test: Read Compiled Tree MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 1212.73 |============= ext4 mdadm raid5 4xNVME ............... 2642.03 |============================= ext4 Crucial P5 Plus 1TB NVME ......... 2747.01 |============================== ZFS zraid1 8xNVME Pool ................ 1217.80 |============= ZFS zraid1 8xNVME Pool no Compression . 1694.75 |=================== ZFS mirror 8xNVME Pool ................ 1454.31 |================ PostMark 1.51 Disk Transaction Performance TPS > Higher Is Better ZFS zraid1 4xNVME Pool ................ 3289 |===================== ext4 mdadm raid5 4xNVME ............... 5068 |================================= ext4 Crucial P5 Plus 1TB NVME ......... 5137 |================================= ZFS zraid1 8xNVME Pool ................ 3275 |===================== ZFS zraid1 8xNVME Pool no Compression . 3275 |===================== ZFS mirror 8xNVME Pool ................ 3318 |=====================