pts-disk-different-nvmes AMD Ryzen Threadripper 1900X 8-Core testing with a Gigabyte X399 DESIGNARE EX-CF (F13a BIOS) and NVIDIA Quadro P400 on Debian 11 via the Phoronix Test Suite. ZFS zraid1 4xNVME Pool: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1920x1080 ext4 mdadm raid5 4xNVME: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1920x1080 ext4 Crucial P5 Plus 1TB NVME: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1920x1080 ZFS zraid1 8xNVME Pool: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1920x1080 ZFS zraid1 8xNVME Pool no Compression: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1920x1080 FS-Mark 3.3 Test: 1000 Files, 1MB Size Files/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 646.5 |================================ ext4 mdadm raid5 4xNVME ............... 254.0 |============= ext4 Crucial P5 Plus 1TB NVME ......... 584.1 |============================= ZFS zraid1 8xNVME Pool ................ 646.7 |================================ ZFS zraid1 8xNVME Pool no Compression . 589.7 |============================= FS-Mark 3.3 Test: 5000 Files, 1MB Size, 4 Threads Files/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 1632.7 |=============================== ext4 mdadm raid5 4xNVME ............... 537.9 |========== ext4 Crucial P5 Plus 1TB NVME ......... 522.1 |========== ZFS zraid1 8xNVME Pool ................ 1655.6 |=============================== ZFS zraid1 8xNVME Pool no Compression . 1240.0 |======================= FS-Mark 3.3 Test: 4000 Files, 32 Sub Dirs, 1MB Size Files/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 684.2 |================================ ext4 mdadm raid5 4xNVME ............... 284.7 |============= ext4 Crucial P5 Plus 1TB NVME ......... 435.0 |==================== ZFS zraid1 8xNVME Pool ................ 644.7 |============================== ZFS zraid1 8xNVME Pool no Compression . 592.1 |============================ FS-Mark 3.3 Test: 1000 Files, 1MB Size, No Sync/FSync Files/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 1359.3 |======================== ext4 mdadm raid5 4xNVME ............... 1733.3 |============================== ext4 Crucial P5 Plus 1TB NVME ......... 1791.6 |=============================== ZFS zraid1 8xNVME Pool ................ 1340.7 |======================= ZFS zraid1 8xNVME Pool no Compression . 1160.9 |==================== Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ................ 2119 |========== ext4 mdadm raid5 4xNVME ............... 6666 |================================= ext4 Crucial P5 Plus 1TB NVME ......... 1765 |========= ZFS zraid1 8xNVME Pool ................ 2015 |========== ZFS zraid1 8xNVME Pool no Compression . 2081 |========== Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ................ 49567 |===== ext4 mdadm raid5 4xNVME ............... 306667 |============================= ext4 Crucial P5 Plus 1TB NVME ......... 329667 |=============================== ZFS zraid1 8xNVME Pool ................ 57433 |===== ZFS zraid1 8xNVME Pool no Compression . 56167 |===== Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ................ 1225 |========================= ext4 mdadm raid5 4xNVME ............... 476 |========== ext4 Crucial P5 Plus 1TB NVME ......... 1623 |================================= ZFS zraid1 8xNVME Pool ................ 1327 |=========================== ZFS zraid1 8xNVME Pool no Compression . 1337 |=========================== Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ................ 41500 |====== ext4 mdadm raid5 4xNVME ............... 82967 |=========== ext4 Crucial P5 Plus 1TB NVME ......... 224667 |=============================== ZFS zraid1 8xNVME Pool ................ 45367 |====== ZFS zraid1 8xNVME Pool no Compression . 48200 |======= Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ................ 2034 |========== ext4 mdadm raid5 4xNVME ............... 6706 |================================= ext4 Crucial P5 Plus 1TB NVME ......... 1765 |========= ZFS zraid1 8xNVME Pool ................ 1999 |========== ZFS zraid1 8xNVME Pool no Compression . 1995 |========== Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ................ 329333 |============================== ext4 mdadm raid5 4xNVME ............... 221333 |==================== ext4 Crucial P5 Plus 1TB NVME ......... 335667 |=============================== ZFS zraid1 8xNVME Pool ................ 326333 |============================== ZFS zraid1 8xNVME Pool no Compression . 331000 |=============================== Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ................ 1029 |===================== ext4 mdadm raid5 4xNVME ............... 650 |============= ext4 Crucial P5 Plus 1TB NVME ......... 1614 |================================= ZFS zraid1 8xNVME Pool ................ 1186 |======================== ZFS zraid1 8xNVME Pool no Compression . 1220 |========================= Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ................ 171333 |======================= ext4 mdadm raid5 4xNVME ............... 115000 |================ ext4 Crucial P5 Plus 1TB NVME ......... 228000 |=============================== ZFS zraid1 8xNVME Pool ................ 170667 |======================= ZFS zraid1 8xNVME Pool no Compression . 174667 |======================== Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 4245 |================================= ext4 Crucial P5 Plus 1TB NVME ......... 3538 |============================ ZFS zraid1 8xNVME Pool ................ 4037 |=============================== ZFS zraid1 8xNVME Pool no Compression . 4170 |================================ Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 194 |===== ext4 mdadm raid5 4xNVME ............... 1199 |=============================== ext4 Crucial P5 Plus 1TB NVME ......... 1286 |================================= ZFS zraid1 8xNVME Pool ................ 224 |====== ZFS zraid1 8xNVME Pool no Compression . 219 |====== Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 2458 |========================= ext4 mdadm raid5 4xNVME ............... 958 |========== ext4 Crucial P5 Plus 1TB NVME ......... 3252 |================================= ZFS zraid1 8xNVME Pool ................ 2661 |=========================== ZFS zraid1 8xNVME Pool no Compression . 2681 |=========================== Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 162 |====== ext4 mdadm raid5 4xNVME ............... 324 |============= ext4 Crucial P5 Plus 1TB NVME ......... 878 |================================== ZFS zraid1 8xNVME Pool ................ 177 |======= ZFS zraid1 8xNVME Pool no Compression . 188 |======= Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 4075 |================================= ext4 Crucial P5 Plus 1TB NVME ......... 3538 |============================= ZFS zraid1 8xNVME Pool ................ 4005 |================================ ZFS zraid1 8xNVME Pool no Compression . 3998 |================================ Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 1287 |================================ ext4 mdadm raid5 4xNVME ............... 865 |====================== ext4 Crucial P5 Plus 1TB NVME ......... 1311 |================================= ZFS zraid1 8xNVME Pool ................ 1275 |================================ ZFS zraid1 8xNVME Pool no Compression . 1294 |================================= Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 2065 |===================== ext4 mdadm raid5 4xNVME ............... 1308 |============= ext4 Crucial P5 Plus 1TB NVME ......... 3236 |================================= ZFS zraid1 8xNVME Pool ................ 2381 |======================== ZFS zraid1 8xNVME Pool no Compression . 2446 |========================= Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 670 |========================== ext4 mdadm raid5 4xNVME ............... 449 |================= ext4 Crucial P5 Plus 1TB NVME ......... 891 |================================== ZFS zraid1 8xNVME Pool ................ 665 |========================= ZFS zraid1 8xNVME Pool no Compression . 682 |========================== Dbench 4.0 12 Clients MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 2626.03 |============================ ext4 mdadm raid5 4xNVME ............... 2484.85 |=========================== ext4 Crucial P5 Plus 1TB NVME ......... 658.79 |======= ZFS zraid1 8xNVME Pool ................ 2472.42 |=========================== ZFS zraid1 8xNVME Pool no Compression . 2783.64 |============================== Dbench 4.0 1 Clients MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 410.17 |============================ ext4 mdadm raid5 4xNVME ............... 453.17 |=============================== ext4 Crucial P5 Plus 1TB NVME ......... 102.99 |======= ZFS zraid1 8xNVME Pool ................ 380.35 |========================== ZFS zraid1 8xNVME Pool no Compression . 458.91 |=============================== Compile Bench 0.6 Test: Compile MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 1398.57 |============================ ext4 mdadm raid5 4xNVME ............... 1478.00 |============================== ext4 Crucial P5 Plus 1TB NVME ......... 1483.07 |============================== ZFS zraid1 8xNVME Pool ................ 1390.71 |============================ ZFS zraid1 8xNVME Pool no Compression . 1247.36 |========================= Compile Bench 0.6 Test: Initial Create MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 222.04 |================ ext4 mdadm raid5 4xNVME ............... 411.07 |============================== ext4 Crucial P5 Plus 1TB NVME ......... 423.67 |=============================== ZFS zraid1 8xNVME Pool ................ 221.83 |================ ZFS zraid1 8xNVME Pool no Compression . 206.77 |=============== Compile Bench 0.6 Test: Read Compiled Tree MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ................ 1212.73 |============= ext4 mdadm raid5 4xNVME ............... 2642.03 |============================= ext4 Crucial P5 Plus 1TB NVME ......... 2747.01 |============================== ZFS zraid1 8xNVME Pool ................ 1217.80 |============= ZFS zraid1 8xNVME Pool no Compression . 1694.75 |=================== PostMark 1.51 Disk Transaction Performance TPS > Higher Is Better ZFS zraid1 4xNVME Pool ................ 3289 |===================== ext4 mdadm raid5 4xNVME ............... 5068 |================================= ext4 Crucial P5 Plus 1TB NVME ......... 5137 |================================= ZFS zraid1 8xNVME Pool ................ 3275 |===================== ZFS zraid1 8xNVME Pool no Compression . 3275 |===================== SQLite 3.30.1 Threads / Copies: 1 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool ................ 7.852 |============================= ext4 mdadm raid5 4xNVME ............... 8.810 |================================ ext4 Crucial P5 Plus 1TB NVME ......... 8.367 |============================== ZFS zraid1 8xNVME Pool ................ 8.380 |============================== ZFS zraid1 8xNVME Pool no Compression . 8.379 |============================== SQLite 3.30.1 Threads / Copies: 8 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool ................ 11.27 |================== ext4 mdadm raid5 4xNVME ............... 20.01 |================================ ext4 Crucial P5 Plus 1TB NVME ......... 15.77 |========================= ZFS zraid1 8xNVME Pool ................ 12.86 |===================== ZFS zraid1 8xNVME Pool no Compression . 12.83 |===================== SQLite 3.30.1 Threads / Copies: 32 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool ................ 23.19 |======================= ext4 mdadm raid5 4xNVME ............... 31.79 |================================ ext4 Crucial P5 Plus 1TB NVME ......... 27.58 |============================ ZFS zraid1 8xNVME Pool ................ 27.00 |=========================== ZFS zraid1 8xNVME Pool no Compression . 26.89 |=========================== SQLite 3.30.1 Threads / Copies: 64 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool ................ 37.45 |========================== ext4 mdadm raid5 4xNVME ............... 45.72 |================================ ext4 Crucial P5 Plus 1TB NVME ......... 40.98 |============================= ZFS zraid1 8xNVME Pool ................ 40.30 |============================ ZFS zraid1 8xNVME Pool no Compression . 40.00 |============================ SQLite 3.30.1 Threads / Copies: 128 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool ................ 76.09 |============================== ext4 mdadm raid5 4xNVME ............... 80.15 |================================ ext4 Crucial P5 Plus 1TB NVME ......... 59.89 |======================== ZFS zraid1 8xNVME Pool ................ 78.15 |=============================== ZFS zraid1 8xNVME Pool no Compression . 77.05 |===============================