pts-disk-different-nvmes AMD Ryzen Threadripper 1900X 8-Core testing with a Gigabyte X399 DESIGNARE EX-CF (F13a BIOS) and NVIDIA Quadro P400 on Debian 11 via the Phoronix Test Suite. ZFS zraid1 4xNVME Pool: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1920x1080 ext4 mdadm raid5 4xNVME: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1920x1080 ext4 Crucial P5 Plus 1TB NVME: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1920x1080 Compile Bench 0.6 Test: Compile MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 1398.57 |==================================== ext4 mdadm raid5 4xNVME ....... 1478.00 |====================================== ext4 Crucial P5 Plus 1TB NVME . 1483.07 |====================================== Compile Bench 0.6 Test: Initial Create MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 222.04 |==================== ext4 mdadm raid5 4xNVME ....... 411.07 |====================================== ext4 Crucial P5 Plus 1TB NVME . 423.67 |======================================= Compile Bench 0.6 Test: Read Compiled Tree MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 1212.73 |================= ext4 mdadm raid5 4xNVME ....... 2642.03 |===================================== ext4 Crucial P5 Plus 1TB NVME . 2747.01 |====================================== IOR 3.3.0 Block Size: 8MB - Disk Target: Default Test Directory MB/s > Higher Is Better Dbench 4.0 12 Clients MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 2626.03 |====================================== ext4 mdadm raid5 4xNVME ....... 2484.85 |==================================== ext4 Crucial P5 Plus 1TB NVME . 658.79 |========== IOR 3.3.0 Block Size: 64MB - Disk Target: Default Test Directory MB/s > Higher Is Better Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 4245 |========================================= ext4 Crucial P5 Plus 1TB NVME . 3538 |================================== Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ........ 2119 |============= ext4 mdadm raid5 4xNVME ....... 6666 |========================================= ext4 Crucial P5 Plus 1TB NVME . 1765 |=========== IOR 3.3.0 Block Size: 16MB - Disk Target: /mnt/mdadm_raid5_4disks MB/s > Higher Is Better Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 194 |====== ext4 mdadm raid5 4xNVME ....... 1199 |====================================== ext4 Crucial P5 Plus 1TB NVME . 1286 |========================================= Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ........ 49567 |====== ext4 mdadm raid5 4xNVME ....... 306667 |==================================== ext4 Crucial P5 Plus 1TB NVME . 329667 |======================================= Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 2458 |=============================== ext4 mdadm raid5 4xNVME ....... 958 |============ ext4 Crucial P5 Plus 1TB NVME . 3252 |========================================= IOR 3.3.0 Block Size: 64MB - Disk Target: /mnt/mdadm_raid5_4disks MB/s > Higher Is Better Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ........ 1225 |=============================== ext4 mdadm raid5 4xNVME ....... 476 |============ ext4 Crucial P5 Plus 1TB NVME . 1623 |========================================= Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 162 |======== ext4 mdadm raid5 4xNVME ....... 324 |=============== ext4 Crucial P5 Plus 1TB NVME . 878 |========================================== Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ........ 41500 |======= ext4 mdadm raid5 4xNVME ....... 82967 |============== ext4 Crucial P5 Plus 1TB NVME . 224667 |======================================= Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 4075 |========================================= ext4 Crucial P5 Plus 1TB NVME . 3538 |==================================== IOR 3.3.0 Block Size: 1024MB - Disk Target: /mnt/mdadm_raid5_4disks MB/s > Higher Is Better Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ........ 2034 |============ ext4 mdadm raid5 4xNVME ....... 6706 |========================================= ext4 Crucial P5 Plus 1TB NVME . 1765 |=========== IOR 3.3.0 Block Size: 16MB - Disk Target: Default Test Directory MB/s > Higher Is Better IOR 3.3.0 Block Size: 32MB - Disk Target: Default Test Directory MB/s > Higher Is Better Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 1287 |======================================== ext4 mdadm raid5 4xNVME ....... 865 |=========================== ext4 Crucial P5 Plus 1TB NVME . 1311 |========================================= Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ........ 329333 |====================================== ext4 mdadm raid5 4xNVME ....... 221333 |========================== ext4 Crucial P5 Plus 1TB NVME . 335667 |======================================= Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 2065 |========================== ext4 mdadm raid5 4xNVME ....... 1308 |================= ext4 Crucial P5 Plus 1TB NVME . 3236 |========================================= Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ........ 1029 |========================== ext4 mdadm raid5 4xNVME ....... 650 |================= ext4 Crucial P5 Plus 1TB NVME . 1614 |========================================= IOR 3.3.0 Block Size: 256MB - Disk Target: /nvme_pool MB/s > Higher Is Better Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 670 |================================ ext4 mdadm raid5 4xNVME ....... 449 |===================== ext4 Crucial P5 Plus 1TB NVME . 891 |========================================== IOR 3.3.0 Block Size: 64MB - Disk Target: /nvme_pool MB/s > Higher Is Better IOR 3.3.0 Block Size: 256MB - Disk Target: /mnt/mdadm_raid5_4disks MB/s > Higher Is Better Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool ........ 171333 |============================= ext4 mdadm raid5 4xNVME ....... 115000 |==================== ext4 Crucial P5 Plus 1TB NVME . 228000 |======================================= FS-Mark 3.3 Test: 1000 Files, 1MB Size Files/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 646.5 |======================================== ext4 mdadm raid5 4xNVME ....... 254.0 |================ ext4 Crucial P5 Plus 1TB NVME . 584.1 |==================================== FS-Mark 3.3 Test: 5000 Files, 1MB Size, 4 Threads Files/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 1632.7 |======================================= ext4 mdadm raid5 4xNVME ....... 537.9 |============= ext4 Crucial P5 Plus 1TB NVME . 522.1 |============ IOR 3.3.0 Block Size: 4MB - Disk Target: /mnt/single_ssd MB/s > Higher Is Better FS-Mark 3.3 Test: 4000 Files, 32 Sub Dirs, 1MB Size Files/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 684.2 |======================================== ext4 mdadm raid5 4xNVME ....... 284.7 |================= ext4 Crucial P5 Plus 1TB NVME . 435.0 |========================= FS-Mark 3.3 Test: 1000 Files, 1MB Size, No Sync/FSync Files/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 1359.3 |============================== ext4 mdadm raid5 4xNVME ....... 1733.3 |====================================== ext4 Crucial P5 Plus 1TB NVME . 1791.6 |======================================= IOR 3.3.0 Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better IOR 3.3.0 Block Size: 4MB - Disk Target: Default Test Directory MB/s > Higher Is Better IOR 3.3.0 Block Size: 512MB - Disk Target: /mnt/single_ssd MB/s > Higher Is Better IOR 3.3.0 Block Size: 1024MB - Disk Target: /mnt/single_ssd MB/s > Higher Is Better IOR 3.3.0 Block Size: 64MB - Disk Target: /mnt/single_ssd MB/s > Higher Is Better IOR 3.3.0 Block Size: 256MB - Disk Target: /mnt/single_ssd MB/s > Higher Is Better IOR 3.3.0 Block Size: 16MB - Disk Target: /mnt/single_ssd MB/s > Higher Is Better IOR 3.3.0 Block Size: 32MB - Disk Target: /mnt/single_ssd MB/s > Higher Is Better IOR 3.3.0 Block Size: 8MB - Disk Target: /mnt/single_ssd MB/s > Higher Is Better IOR 3.3.0 Block Size: 32MB - Disk Target: /mnt/mdadm_raid5_4disks MB/s > Higher Is Better IOR 3.3.0 Block Size: 2MB - Disk Target: /mnt/single_ssd MB/s > Higher Is Better IOR 3.3.0 Block Size: 1024MB - Disk Target: /nvme_pool MB/s > Higher Is Better IOzone 3.465 8GB Write Performance MB/s > Higher Is Better IOR 3.3.0 Block Size: 2MB - Disk Target: /mnt/mdadm_raid5_4disks MB/s > Higher Is Better Dbench 4.0 1 Clients MB/s > Higher Is Better ZFS zraid1 4xNVME Pool ........ 410.17 |=================================== ext4 mdadm raid5 4xNVME ....... 453.17 |======================================= ext4 Crucial P5 Plus 1TB NVME . 102.99 |========= IOR 3.3.0 Block Size: 4MB - Disk Target: /mnt/mdadm_raid5_4disks MB/s > Higher Is Better IOR 3.3.0 Block Size: 2MB - Disk Target: /nvme_pool MB/s > Higher Is Better IOR 3.3.0 Block Size: 8MB - Disk Target: /mnt/mdadm_raid5_4disks MB/s > Higher Is Better IOR 3.3.0 Block Size: 4MB - Disk Target: /nvme_pool MB/s > Higher Is Better IOR 3.3.0 Block Size: 256MB - Disk Target: Default Test Directory MB/s > Higher Is Better IOR 3.3.0 Block Size: 8MB - Disk Target: /nvme_pool MB/s > Higher Is Better IOR 3.3.0 Block Size: 512MB - Disk Target: Default Test Directory MB/s > Higher Is Better IOR 3.3.0 Block Size: 16MB - Disk Target: /nvme_pool MB/s > Higher Is Better IOR 3.3.0 Block Size: 1024MB - Disk Target: Default Test Directory MB/s > Higher Is Better IOR 3.3.0 Block Size: 32MB - Disk Target: /nvme_pool MB/s > Higher Is Better IOR 3.3.0 Block Size: 512MB - Disk Target: /mnt/mdadm_raid5_4disks MB/s > Higher Is Better IOR 3.3.0 Block Size: 512MB - Disk Target: /nvme_pool MB/s > Higher Is Better PostMark 1.51 Disk Transaction Performance TPS > Higher Is Better ZFS zraid1 4xNVME Pool ........ 3289 |========================== ext4 mdadm raid5 4xNVME ....... 5068 |======================================== ext4 Crucial P5 Plus 1TB NVME . 5137 |========================================= SQLite 3.30.1 Threads / Copies: 1 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool ........ 7.852 |==================================== ext4 mdadm raid5 4xNVME ....... 8.810 |======================================== ext4 Crucial P5 Plus 1TB NVME . 8.367 |====================================== SQLite 3.30.1 Threads / Copies: 8 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool ........ 11.27 |======================= ext4 mdadm raid5 4xNVME ....... 20.01 |======================================== ext4 Crucial P5 Plus 1TB NVME . 15.77 |================================ SQLite 3.30.1 Threads / Copies: 32 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool ........ 23.19 |============================= ext4 mdadm raid5 4xNVME ....... 31.79 |======================================== ext4 Crucial P5 Plus 1TB NVME . 27.58 |=================================== SQLite 3.30.1 Threads / Copies: 64 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool ........ 37.45 |================================= ext4 mdadm raid5 4xNVME ....... 45.72 |======================================== ext4 Crucial P5 Plus 1TB NVME . 40.98 |==================================== SQLite 3.30.1 Threads / Copies: 128 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool ........ 76.09 |====================================== ext4 mdadm raid5 4xNVME ....... 80.15 |======================================== ext4 Crucial P5 Plus 1TB NVME . 59.89 |==============================