pts-disk-different-nvmes AMD Ryzen Threadripper 1900X 8-Core testing with a Gigabyte X399 DESIGNARE EX-CF (F13a BIOS) and NVIDIA Quadro P400 on Debian 11 via the Phoronix Test Suite. ZFS zraid1 4xNVME Pool: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1920x1080 ext4 mdadm raid5 4xNVME: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1920x1080 SQLite 3.30.1 Threads / Copies: 1 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool .. 7.852 |========================================= ext4 mdadm raid5 4xNVME . 8.810 |============================================== SQLite 3.30.1 Threads / Copies: 8 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool .. 11.27 |========================== ext4 mdadm raid5 4xNVME . 20.01 |============================================== SQLite 3.30.1 Threads / Copies: 32 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool .. 23.19 |================================== ext4 mdadm raid5 4xNVME . 31.79 |============================================== SQLite 3.30.1 Threads / Copies: 64 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool .. 37.45 |====================================== ext4 mdadm raid5 4xNVME . 45.72 |============================================== SQLite 3.30.1 Threads / Copies: 128 Seconds < Lower Is Better ZFS zraid1 4xNVME Pool .. 76.09 |============================================ ext4 mdadm raid5 4xNVME . 80.15 |============================================== Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool . 4245 |================================================ Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool .. 2119 |=============== ext4 mdadm raid5 4xNVME . 6666 |=============================================== Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool .. 194 |======== ext4 mdadm raid5 4xNVME . 1199 |=============================================== Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool .. 49567 |======= ext4 mdadm raid5 4xNVME . 306667 |============================================= Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool .. 2458 |=============================================== ext4 mdadm raid5 4xNVME . 958 |================== Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool .. 1225 |=============================================== ext4 mdadm raid5 4xNVME . 476 |================== Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool .. 162 |======================== ext4 mdadm raid5 4xNVME . 324 |================================================ Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool .. 41500 |======================= ext4 mdadm raid5 4xNVME . 82967 |============================================== Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool . 4075 |================================================ Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool .. 2034 |============== ext4 mdadm raid5 4xNVME . 6706 |=============================================== Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool .. 1287 |=============================================== ext4 mdadm raid5 4xNVME . 865 |================================ Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool .. 329333 |============================================= ext4 mdadm raid5 4xNVME . 221333 |============================== Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool .. 2065 |=============================================== ext4 mdadm raid5 4xNVME . 1308 |============================== Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool .. 1029 |=============================================== ext4 mdadm raid5 4xNVME . 650 |============================== Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS zraid1 4xNVME Pool .. 670 |================================================ ext4 mdadm raid5 4xNVME . 449 |================================ Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS zraid1 4xNVME Pool .. 171333 |============================================= ext4 mdadm raid5 4xNVME . 115000 |============================== FS-Mark 3.3 Test: 1000 Files, 1MB Size Files/s > Higher Is Better ZFS zraid1 4xNVME Pool .. 646.5 |============================================== ext4 mdadm raid5 4xNVME . 254.0 |================== FS-Mark 3.3 Test: 5000 Files, 1MB Size, 4 Threads Files/s > Higher Is Better ZFS zraid1 4xNVME Pool .. 1632.7 |============================================= ext4 mdadm raid5 4xNVME . 537.9 |=============== FS-Mark 3.3 Test: 4000 Files, 32 Sub Dirs, 1MB Size Files/s > Higher Is Better ZFS zraid1 4xNVME Pool .. 684.2 |============================================== ext4 mdadm raid5 4xNVME . 284.7 |=================== FS-Mark 3.3 Test: 1000 Files, 1MB Size, No Sync/FSync Files/s > Higher Is Better ZFS zraid1 4xNVME Pool .. 1359.3 |=================================== ext4 mdadm raid5 4xNVME . 1733.3 |============================================= Dbench 4.0 12 Clients MB/s > Higher Is Better ZFS zraid1 4xNVME Pool .. 2626.03 |============================================ ext4 mdadm raid5 4xNVME . 2484.85 |========================================== Dbench 4.0 1 Clients MB/s > Higher Is Better ZFS zraid1 4xNVME Pool .. 410.17 |========================================= ext4 mdadm raid5 4xNVME . 453.17 |============================================= IOR 3.3.0 Block Size: 2MB - Disk Target: /nvme_pool MB/s > Higher Is Better IOR 3.3.0 Block Size: 4MB - Disk Target: /nvme_pool MB/s > Higher Is Better IOR 3.3.0 Block Size: 8MB - Disk Target: /nvme_pool MB/s > Higher Is Better IOR 3.3.0 Block Size: 16MB - Disk Target: /nvme_pool MB/s > Higher Is Better IOR 3.3.0 Block Size: 32MB - Disk Target: /nvme_pool MB/s > Higher Is Better IOR 3.3.0 Block Size: 64MB - Disk Target: /nvme_pool MB/s > Higher Is Better IOR 3.3.0 Block Size: 256MB - Disk Target: /nvme_pool MB/s > Higher Is Better IOR 3.3.0 Block Size: 512MB - Disk Target: /nvme_pool MB/s > Higher Is Better IOR 3.3.0 Block Size: 1024MB - Disk Target: /nvme_pool MB/s > Higher Is Better IOR 3.3.0 Block Size: 2MB - Disk Target: /mnt/single_ssd MB/s > Higher Is Better IOR 3.3.0 Block Size: 4MB - Disk Target: /mnt/single_ssd MB/s > Higher Is Better IOR 3.3.0 Block Size: 8MB - Disk Target: /mnt/single_ssd MB/s > Higher Is Better IOR 3.3.0 Block Size: 16MB - Disk Target: /mnt/single_ssd MB/s > Higher Is Better IOR 3.3.0 Block Size: 32MB - Disk Target: /mnt/single_ssd MB/s > Higher Is Better IOR 3.3.0 Block Size: 64MB - Disk Target: /mnt/single_ssd MB/s > Higher Is Better IOR 3.3.0 Block Size: 256MB - Disk Target: /mnt/single_ssd MB/s > Higher Is Better IOR 3.3.0 Block Size: 512MB - Disk Target: /mnt/single_ssd MB/s > Higher Is Better IOR 3.3.0 Block Size: 1024MB - Disk Target: /mnt/single_ssd MB/s > Higher Is Better IOR 3.3.0 Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better IOR 3.3.0 Block Size: 4MB - Disk Target: Default Test Directory MB/s > Higher Is Better IOR 3.3.0 Block Size: 8MB - Disk Target: Default Test Directory MB/s > Higher Is Better IOR 3.3.0 Block Size: 16MB - Disk Target: Default Test Directory MB/s > Higher Is Better IOR 3.3.0 Block Size: 2MB - Disk Target: /mnt/mdadm_raid5_4disks MB/s > Higher Is Better IOR 3.3.0 Block Size: 32MB - Disk Target: Default Test Directory MB/s > Higher Is Better IOR 3.3.0 Block Size: 4MB - Disk Target: /mnt/mdadm_raid5_4disks MB/s > Higher Is Better IOR 3.3.0 Block Size: 64MB - Disk Target: Default Test Directory MB/s > Higher Is Better IOR 3.3.0 Block Size: 8MB - Disk Target: /mnt/mdadm_raid5_4disks MB/s > Higher Is Better IOR 3.3.0 Block Size: 16MB - Disk Target: /mnt/mdadm_raid5_4disks MB/s > Higher Is Better IOR 3.3.0 Block Size: 256MB - Disk Target: Default Test Directory MB/s > Higher Is Better IOR 3.3.0 Block Size: 32MB - Disk Target: /mnt/mdadm_raid5_4disks MB/s > Higher Is Better IOR 3.3.0 Block Size: 512MB - Disk Target: Default Test Directory MB/s > Higher Is Better IOR 3.3.0 Block Size: 64MB - Disk Target: /mnt/mdadm_raid5_4disks MB/s > Higher Is Better IOR 3.3.0 Block Size: 1024MB - Disk Target: Default Test Directory MB/s > Higher Is Better IOR 3.3.0 Block Size: 256MB - Disk Target: /mnt/mdadm_raid5_4disks MB/s > Higher Is Better IOR 3.3.0 Block Size: 512MB - Disk Target: /mnt/mdadm_raid5_4disks MB/s > Higher Is Better IOR 3.3.0 Block Size: 1024MB - Disk Target: /mnt/mdadm_raid5_4disks MB/s > Higher Is Better IOzone 3.465 8GB Write Performance MB/s > Higher Is Better Compile Bench 0.6 Test: Compile MB/s > Higher Is Better ZFS zraid1 4xNVME Pool .. 1398.57 |========================================== ext4 mdadm raid5 4xNVME . 1478.00 |============================================ Compile Bench 0.6 Test: Initial Create MB/s > Higher Is Better ZFS zraid1 4xNVME Pool .. 222.04 |======================== ext4 mdadm raid5 4xNVME . 411.07 |============================================= Compile Bench 0.6 Test: Read Compiled Tree MB/s > Higher Is Better ZFS zraid1 4xNVME Pool .. 1212.73 |==================== ext4 mdadm raid5 4xNVME . 2642.03 |============================================ PostMark 1.51 Disk Transaction Performance TPS > Higher Is Better ZFS zraid1 4xNVME Pool .. 3289 |=============================== ext4 mdadm raid5 4xNVME . 5068 |===============================================