pts-disk-different-nvmes AMD Ryzen Threadripper 1900X 8-Core testing with a Gigabyte X399 DESIGNARE EX-CF (F13a BIOS) and NVIDIA Quadro P400 on Debian 11 via the Phoronix Test Suite. ZFS zraid1 4xNVME Pool: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1920x1080 ext4 mdadm raid5 4xNVME: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1920x1080 ext4 Crucial P5 Plus 1TB NVME: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1920x1080 ZFS zraid1 8xNVME Pool: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1920x1080 Dbench 4.0 12 Clients MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 2626.03 |====================================== ext4 mdadm raid5 4xNVME ....... 2484.85 |==================================== ext4 Crucial P5 Plus 1TB NVME . 658.79 |========== ZFS zraid1 8xNVME Pool ........ 2472.42 |==================================== Dbench 4.0 1 Clients MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 410.17 |=================================== ext4 mdadm raid5 4xNVME ....... 453.17 |======================================= ext4 Crucial P5 Plus 1TB NVME . 102.99 |========= ZFS zraid1 8xNVME Pool ........ 380.35 |================================= FS-Mark 3.3 Test: 5000 Files, 1MB Size, 4 Threads Files/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 1632.7 |====================================== ext4 mdadm raid5 4xNVME ....... 537.9 |============= ext4 Crucial P5 Plus 1TB NVME . 522.1 |============ ZFS zraid1 8xNVME Pool ........ 1655.6 |======================================= FS-Mark 3.3 Test: 4000 Files, 32 Sub Dirs, 1MB Size Files/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 684.2 |======================================== ext4 mdadm raid5 4xNVME ....... 284.7 |================= ext4 Crucial P5 Plus 1TB NVME . 435.0 |========================= ZFS zraid1 8xNVME Pool ........ 644.7 |====================================== FS-Mark 3.3 Test: 1000 Files, 1MB Size, No Sync/FSync Files/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 1359.3 |============================== ext4 mdadm raid5 4xNVME ....... 1733.3 |====================================== ext4 Crucial P5 Plus 1TB NVME . 1791.6 |======================================= ZFS zraid1 8xNVME Pool ........ 1340.7 |============================= FS-Mark 3.3 Test: 1000 Files, 1MB Size Files/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 646.5 |======================================== ext4 mdadm raid5 4xNVME ....... 254.0 |================ ext4 Crucial P5 Plus 1TB NVME . 584.1 |==================================== ZFS zraid1 8xNVME Pool ........ 646.7 |======================================== SQLite 3.30.1 Threads / Copies: 128 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool ........ 76.09 |====================================== ext4 mdadm raid5 4xNVME ....... 80.15 |======================================== ext4 Crucial P5 Plus 1TB NVME . 59.89 |============================== ZFS zraid1 8xNVME Pool ........ 78.15 |======================================= PostMark 1.51 Disk Transaction Performance TPS > Higher Is Better ZFS zraid1 4xNVME Pool ........ 3289 |========================== ext4 mdadm raid5 4xNVME ....... 5068 |======================================== ext4 Crucial P5 Plus 1TB NVME . 5137 |========================================= ZFS zraid1 8xNVME Pool ........ 3275 |========================== SQLite 3.30.1 Threads / Copies: 64 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool ........ 37.45 |================================= ext4 mdadm raid5 4xNVME ....... 45.72 |======================================== ext4 Crucial P5 Plus 1TB NVME . 40.98 |==================================== ZFS zraid1 8xNVME Pool ........ 40.30 |=================================== Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ........ 329333 |====================================== ext4 mdadm raid5 4xNVME ....... 221333 |========================== ext4 Crucial P5 Plus 1TB NVME . 335667 |======================================= ZFS zraid1 8xNVME Pool ........ 326333 |====================================== Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 1287 |======================================== ext4 mdadm raid5 4xNVME ....... 865 |=========================== ext4 Crucial P5 Plus 1TB NVME . 1311 |========================================= ZFS zraid1 8xNVME Pool ........ 1275 |======================================== Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ........ 49567 |====== ext4 mdadm raid5 4xNVME ....... 306667 |==================================== ext4 Crucial P5 Plus 1TB NVME . 329667 |======================================= ZFS zraid1 8xNVME Pool ........ 57433 |======= Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 194 |====== ext4 mdadm raid5 4xNVME ....... 1199 |====================================== ext4 Crucial P5 Plus 1TB NVME . 1286 |========================================= ZFS zraid1 8xNVME Pool ........ 224 |======= Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ........ 2119 |============= ext4 mdadm raid5 4xNVME ....... 6666 |========================================= ext4 Crucial P5 Plus 1TB NVME . 1765 |=========== ZFS zraid1 8xNVME Pool ........ 2015 |============ Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ........ 2034 |============ ext4 mdadm raid5 4xNVME ....... 6706 |========================================= ext4 Crucial P5 Plus 1TB NVME . 1765 |=========== ZFS zraid1 8xNVME Pool ........ 1999 |============ Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 4245 |========================================= ext4 Crucial P5 Plus 1TB NVME . 3538 |================================== ZFS zraid1 8xNVME Pool ........ 4037 |======================================= Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 4075 |========================================= ext4 Crucial P5 Plus 1TB NVME . 3538 |==================================== ZFS zraid1 8xNVME Pool ........ 4005 |======================================== Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ........ 1225 |=============================== ext4 mdadm raid5 4xNVME ....... 476 |============ ext4 Crucial P5 Plus 1TB NVME . 1623 |========================================= ZFS zraid1 8xNVME Pool ........ 1327 |================================== Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 2458 |=============================== ext4 mdadm raid5 4xNVME ....... 958 |============ ext4 Crucial P5 Plus 1TB NVME . 3252 |========================================= ZFS zraid1 8xNVME Pool ........ 2661 |================================== Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ........ 1029 |========================== ext4 mdadm raid5 4xNVME ....... 650 |================= ext4 Crucial P5 Plus 1TB NVME . 1614 |========================================= ZFS zraid1 8xNVME Pool ........ 1186 |============================== Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 2065 |========================== ext4 mdadm raid5 4xNVME ....... 1308 |================= ext4 Crucial P5 Plus 1TB NVME . 3236 |========================================= ZFS zraid1 8xNVME Pool ........ 2381 |============================== Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ........ 41500 |======= ext4 mdadm raid5 4xNVME ....... 82967 |============== ext4 Crucial P5 Plus 1TB NVME . 224667 |======================================= ZFS zraid1 8xNVME Pool ........ 45367 |======== Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 162 |======== ext4 mdadm raid5 4xNVME ....... 324 |=============== ext4 Crucial P5 Plus 1TB NVME . 878 |========================================== ZFS zraid1 8xNVME Pool ........ 177 |======== Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ........ 171333 |============================= ext4 mdadm raid5 4xNVME ....... 115000 |==================== ext4 Crucial P5 Plus 1TB NVME . 228000 |======================================= ZFS zraid1 8xNVME Pool ........ 170667 |============================= Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 670 |================================ ext4 mdadm raid5 4xNVME ....... 449 |===================== ext4 Crucial P5 Plus 1TB NVME . 891 |========================================== ZFS zraid1 8xNVME Pool ........ 665 |=============================== Compile Bench 0.6 Test: Compile MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 1398.57 |==================================== ext4 mdadm raid5 4xNVME ....... 1478.00 |====================================== ext4 Crucial P5 Plus 1TB NVME . 1483.07 |====================================== ZFS zraid1 8xNVME Pool ........ 1390.71 |==================================== SQLite 3.30.1 Threads / Copies: 32 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool ........ 23.19 |============================= ext4 mdadm raid5 4xNVME ....... 31.79 |======================================== ext4 Crucial P5 Plus 1TB NVME . 27.58 |=================================== ZFS zraid1 8xNVME Pool ........ 27.00 |================================== SQLite 3.30.1 Threads / Copies: 1 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool ........ 7.852 |==================================== ext4 mdadm raid5 4xNVME ....... 8.810 |======================================== ext4 Crucial P5 Plus 1TB NVME . 8.367 |====================================== ZFS zraid1 8xNVME Pool ........ 8.380 |====================================== SQLite 3.30.1 Threads / Copies: 8 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool ........ 11.27 |======================= ext4 mdadm raid5 4xNVME ....... 20.01 |======================================== ext4 Crucial P5 Plus 1TB NVME . 15.77 |================================ ZFS zraid1 8xNVME Pool ........ 12.86 |========================== Compile Bench 0.6 Test: Read Compiled Tree MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 1212.73 |================= ext4 mdadm raid5 4xNVME ....... 2642.03 |===================================== ext4 Crucial P5 Plus 1TB NVME . 2747.01 |====================================== ZFS zraid1 8xNVME Pool ........ 1217.80 |================= Compile Bench 0.6 Test: Initial Create MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 222.04 |==================== ext4 mdadm raid5 4xNVME ....... 411.07 |====================================== ext4 Crucial P5 Plus 1TB NVME . 423.67 |======================================= ZFS zraid1 8xNVME Pool ........ 221.83 |====================