pts-disk-different-nvmes AMD Ryzen Threadripper 1900X 8-Core testing with a Gigabyte X399 DESIGNARE EX-CF (F13a BIOS) and NVIDIA Quadro P400 on Debian 11 via the Phoronix Test Suite. ZFS raidz1 4xNVME: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1920x1080 ext4 soft raid5 4xNVME: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1920x1080 ext4 Crucial P5 Plus 1TB NVME: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1920x1080 ZFS raidz1 8xNVME: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1920x1080 ZFS raidz1 8xNVME no Compression: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1920x1080 ZFS mirror 8xNVME: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1920x1080 ext4 soft raid5 8xNVME: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1920x1080 SQLite 3.30.1 Threads / Copies: 1 Seconds < Lower Is Better ZFS raidz1 4xNVME ................ 7.852 |=========================== ext4 soft raid5 4xNVME ........... 8.810 |============================== ext4 Crucial P5 Plus 1TB NVME .... 8.367 |============================= ZFS raidz1 8xNVME ................ 8.380 |============================= ZFS raidz1 8xNVME no Compression . 8.379 |============================= ZFS mirror 8xNVME ................ 7.252 |========================= ext4 soft raid5 8xNVME ........... 10.426 |==================================== SQLite 3.30.1 Threads / Copies: 8 Seconds < Lower Is Better ZFS raidz1 4xNVME ................ 11.267 |================== ext4 soft raid5 4xNVME ........... 20.005 |================================ ext4 Crucial P5 Plus 1TB NVME .... 15.767 |========================= ZFS raidz1 8xNVME ................ 12.857 |==================== ZFS raidz1 8xNVME no Compression . 12.827 |==================== ZFS mirror 8xNVME ................ 9.043 |============== ext4 soft raid5 8xNVME ........... 22.829 |==================================== SQLite 3.30.1 Threads / Copies: 32 Seconds < Lower Is Better ZFS raidz1 4xNVME ................ 23.19 |======================== ext4 soft raid5 4xNVME ........... 31.79 |================================= ext4 Crucial P5 Plus 1TB NVME .... 27.58 |============================ ZFS raidz1 8xNVME ................ 27.00 |============================ ZFS raidz1 8xNVME no Compression . 26.89 |============================ ZFS mirror 8xNVME ................ 17.74 |================== ext4 soft raid5 8xNVME ........... 35.99 |===================================== SQLite 3.30.1 Threads / Copies: 64 Seconds < Lower Is Better ZFS raidz1 4xNVME ................ 37.45 |============================ ext4 soft raid5 4xNVME ........... 45.72 |================================== ext4 Crucial P5 Plus 1TB NVME .... 40.98 |============================== ZFS raidz1 8xNVME ................ 40.30 |============================== ZFS raidz1 8xNVME no Compression . 40.00 |============================= ZFS mirror 8xNVME ................ 36.60 |=========================== ext4 soft raid5 8xNVME ........... 50.31 |===================================== SQLite 3.30.1 Threads / Copies: 128 Seconds < Lower Is Better ZFS raidz1 4xNVME ................ 76.09 |================================== ext4 soft raid5 4xNVME ........... 80.15 |=================================== ext4 Crucial P5 Plus 1TB NVME .... 59.89 |========================== ZFS raidz1 8xNVME ................ 78.15 |================================== ZFS raidz1 8xNVME no Compression . 77.05 |================================== ZFS mirror 8xNVME ................ 77.88 |================================== ext4 soft raid5 8xNVME ........... 83.86 |===================================== Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 4245 |======================== ext4 Crucial P5 Plus 1TB NVME .... 3538 |==================== ZFS raidz1 8xNVME ................ 4037 |======================= ZFS raidz1 8xNVME no Compression . 4170 |======================== ZFS mirror 8xNVME ................ 6617 |====================================== Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS raidz1 4xNVME ................ 2119 |====== ext4 soft raid5 4xNVME ........... 6666 |=================== ext4 Crucial P5 Plus 1TB NVME .... 1765 |===== ZFS raidz1 8xNVME ................ 2015 |====== ZFS raidz1 8xNVME no Compression . 2081 |====== ZFS mirror 8xNVME ................ 3305 |========= ext4 soft raid5 8xNVME ........... 13225 |===================================== Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 194 |====== ext4 soft raid5 4xNVME ........... 1199 |=================================== ext4 Crucial P5 Plus 1TB NVME .... 1286 |====================================== ZFS raidz1 8xNVME ................ 224 |======= ZFS raidz1 8xNVME no Compression . 219 |====== ZFS mirror 8xNVME ................ 877 |========================== ext4 soft raid5 8xNVME ........... 1065 |=============================== Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS raidz1 4xNVME ................ 49567 |===== ext4 soft raid5 4xNVME ........... 306667 |================================= ext4 Crucial P5 Plus 1TB NVME .... 329667 |==================================== ZFS raidz1 8xNVME ................ 57433 |====== ZFS raidz1 8xNVME no Compression . 56167 |====== ZFS mirror 8xNVME ................ 224333 |======================== ext4 soft raid5 8xNVME ........... 272667 |============================== Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 2458 |========================== ext4 soft raid5 4xNVME ........... 958 |========== ext4 Crucial P5 Plus 1TB NVME .... 3252 |=================================== ZFS raidz1 8xNVME ................ 2661 |============================ ZFS raidz1 8xNVME no Compression . 2681 |============================= ZFS mirror 8xNVME ................ 3573 |====================================== ext4 soft raid5 8xNVME ........... 1032 |=========== Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS raidz1 4xNVME ................ 1225 |========================== ext4 soft raid5 4xNVME ........... 476 |========== ext4 Crucial P5 Plus 1TB NVME .... 1623 |=================================== ZFS raidz1 8xNVME ................ 1327 |============================ ZFS raidz1 8xNVME no Compression . 1337 |============================ ZFS mirror 8xNVME ................ 1783 |====================================== ext4 soft raid5 8xNVME ........... 512 |=========== Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 162 |======= ext4 soft raid5 4xNVME ........... 324 |============== ext4 Crucial P5 Plus 1TB NVME .... 878 |======================================= ZFS raidz1 8xNVME ................ 177 |======== ZFS raidz1 8xNVME no Compression . 188 |======== ZFS mirror 8xNVME ................ 267 |============ ext4 soft raid5 8xNVME ........... 293 |============= Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS raidz1 4xNVME ................ 41500 |======= ext4 soft raid5 4xNVME ........... 82967 |============= ext4 Crucial P5 Plus 1TB NVME .... 224667 |==================================== ZFS raidz1 8xNVME ................ 45367 |======= ZFS raidz1 8xNVME no Compression . 48200 |======== ZFS mirror 8xNVME ................ 68333 |=========== ext4 soft raid5 8xNVME ........... 75067 |============ Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 4075 |===================================== ext4 Crucial P5 Plus 1TB NVME .... 3538 |================================ ZFS raidz1 8xNVME ................ 4005 |===================================== ZFS raidz1 8xNVME no Compression . 3998 |===================================== ZFS mirror 8xNVME ................ 4154 |====================================== Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS raidz1 4xNVME ................ 2034 |====== ext4 soft raid5 4xNVME ........... 6706 |=================== ext4 Crucial P5 Plus 1TB NVME .... 1765 |===== ZFS raidz1 8xNVME ................ 1999 |====== ZFS raidz1 8xNVME no Compression . 1995 |====== ZFS mirror 8xNVME ................ 2073 |====== ext4 soft raid5 8xNVME ........... 13200 |===================================== Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 1287 |===================================== ext4 soft raid5 4xNVME ........... 865 |========================= ext4 Crucial P5 Plus 1TB NVME .... 1311 |====================================== ZFS raidz1 8xNVME ................ 1275 |===================================== ZFS raidz1 8xNVME no Compression . 1294 |====================================== ZFS mirror 8xNVME ................ 1296 |====================================== ext4 soft raid5 8xNVME ........... 779 |======================= Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS raidz1 4xNVME ................ 329333 |=================================== ext4 soft raid5 4xNVME ........... 221333 |======================== ext4 Crucial P5 Plus 1TB NVME .... 335667 |==================================== ZFS raidz1 8xNVME ................ 326333 |=================================== ZFS raidz1 8xNVME no Compression . 331000 |=================================== ZFS mirror 8xNVME ................ 332000 |==================================== ext4 soft raid5 8xNVME ........... 199667 |===================== Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 2065 |====================== ext4 soft raid5 4xNVME ........... 1308 |============== ext4 Crucial P5 Plus 1TB NVME .... 3236 |=================================== ZFS raidz1 8xNVME ................ 2381 |========================= ZFS raidz1 8xNVME no Compression . 2446 |========================== ZFS mirror 8xNVME ................ 3555 |====================================== ext4 soft raid5 8xNVME ........... 1636 |================= Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS raidz1 4xNVME ................ 1029 |====================== ext4 soft raid5 4xNVME ........... 650 |============== ext4 Crucial P5 Plus 1TB NVME .... 1614 |=================================== ZFS raidz1 8xNVME ................ 1186 |========================= ZFS raidz1 8xNVME no Compression . 1220 |========================== ZFS mirror 8xNVME ................ 1774 |====================================== ext4 soft raid5 8xNVME ........... 814 |================= Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 670 |============================= ext4 soft raid5 4xNVME ........... 449 |==================== ext4 Crucial P5 Plus 1TB NVME .... 891 |======================================= ZFS raidz1 8xNVME ................ 665 |============================= ZFS raidz1 8xNVME no Compression . 682 |============================== ZFS mirror 8xNVME ................ 771 |================================== ext4 soft raid5 8xNVME ........... 387 |================= Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS raidz1 4xNVME ................ 171333 |=========================== ext4 soft raid5 4xNVME ........... 115000 |================== ext4 Crucial P5 Plus 1TB NVME .... 228000 |==================================== ZFS raidz1 8xNVME ................ 170667 |=========================== ZFS raidz1 8xNVME no Compression . 174667 |============================ ZFS mirror 8xNVME ................ 197333 |=============================== ext4 soft raid5 8xNVME ........... 99067 |================ FS-Mark 3.3 Test: 1000 Files, 1MB Size Files/s > Higher Is Better ZFS raidz1 4xNVME ................ 646.5 |===================================== ext4 soft raid5 4xNVME ........... 254.0 |=============== ext4 Crucial P5 Plus 1TB NVME .... 584.1 |================================= ZFS raidz1 8xNVME ................ 646.7 |===================================== ZFS raidz1 8xNVME no Compression . 589.7 |================================== ZFS mirror 8xNVME ................ 581.1 |================================= ext4 soft raid5 8xNVME ........... 264.4 |=============== FS-Mark 3.3 Test: 5000 Files, 1MB Size, 4 Threads Files/s > Higher Is Better ZFS raidz1 4xNVME ................ 1632.7 |================================== ext4 soft raid5 4xNVME ........... 537.9 |=========== ext4 Crucial P5 Plus 1TB NVME .... 522.1 |=========== ZFS raidz1 8xNVME ................ 1655.6 |=================================== ZFS raidz1 8xNVME no Compression . 1240.0 |========================== ZFS mirror 8xNVME ................ 1711.4 |==================================== ext4 soft raid5 8xNVME ........... 570.0 |============ FS-Mark 3.3 Test: 4000 Files, 32 Sub Dirs, 1MB Size Files/s > Higher Is Better ZFS raidz1 4xNVME ................ 684.2 |===================================== ext4 soft raid5 4xNVME ........... 284.7 |=============== ext4 Crucial P5 Plus 1TB NVME .... 435.0 |======================== ZFS raidz1 8xNVME ................ 644.7 |=================================== ZFS raidz1 8xNVME no Compression . 592.1 |================================ ZFS mirror 8xNVME ................ 658.2 |==================================== ext4 soft raid5 8xNVME ........... 286.6 |=============== FS-Mark 3.3 Test: 1000 Files, 1MB Size, No Sync/FSync Files/s > Higher Is Better ZFS raidz1 4xNVME ................ 1359.3 |=========================== ext4 soft raid5 4xNVME ........... 1733.3 |=================================== ext4 Crucial P5 Plus 1TB NVME .... 1791.6 |==================================== ZFS raidz1 8xNVME ................ 1340.7 |=========================== ZFS raidz1 8xNVME no Compression . 1160.9 |======================= ZFS mirror 8xNVME ................ 1338.4 |=========================== ext4 soft raid5 8xNVME ........... 1782.9 |==================================== Dbench 4.0 12 Clients MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 2626.03 |================================= ext4 soft raid5 4xNVME ........... 2484.85 |=============================== ext4 Crucial P5 Plus 1TB NVME .... 658.79 |======== ZFS raidz1 8xNVME ................ 2472.42 |=============================== ZFS raidz1 8xNVME no Compression . 2783.64 |=================================== ZFS mirror 8xNVME ................ 2730.75 |================================== ext4 soft raid5 8xNVME ........... 2330.16 |============================= Dbench 4.0 1 Clients MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 410.17 |================================ ext4 soft raid5 4xNVME ........... 453.17 |==================================== ext4 Crucial P5 Plus 1TB NVME .... 102.99 |======== ZFS raidz1 8xNVME ................ 380.35 |============================== ZFS raidz1 8xNVME no Compression . 458.91 |==================================== ZFS mirror 8xNVME ................ 435.70 |================================== ext4 soft raid5 8xNVME ........... 423.83 |================================= Compile Bench 0.6 Test: Compile MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 1398.57 |================================= ext4 soft raid5 4xNVME ........... 1478.00 |=================================== ext4 Crucial P5 Plus 1TB NVME .... 1483.07 |=================================== ZFS raidz1 8xNVME ................ 1390.71 |================================= ZFS raidz1 8xNVME no Compression . 1247.36 |============================= ZFS mirror 8xNVME ................ 1408.42 |================================= ext4 soft raid5 8xNVME ........... 1474.80 |=================================== Compile Bench 0.6 Test: Initial Create MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 222.04 |=================== ext4 soft raid5 4xNVME ........... 411.07 |=================================== ext4 Crucial P5 Plus 1TB NVME .... 423.67 |==================================== ZFS raidz1 8xNVME ................ 221.83 |=================== ZFS raidz1 8xNVME no Compression . 206.77 |================== ZFS mirror 8xNVME ................ 216.21 |================== ext4 soft raid5 8xNVME ........... 413.46 |=================================== Compile Bench 0.6 Test: Read Compiled Tree MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 1212.73 |=============== ext4 soft raid5 4xNVME ........... 2642.03 |================================== ext4 Crucial P5 Plus 1TB NVME .... 2747.01 |=================================== ZFS raidz1 8xNVME ................ 1217.80 |================ ZFS raidz1 8xNVME no Compression . 1694.75 |====================== ZFS mirror 8xNVME ................ 1454.31 |=================== ext4 soft raid5 8xNVME ........... 2683.76 |================================== PostMark 1.51 Disk Transaction Performance TPS > Higher Is Better ZFS raidz1 4xNVME ................ 3289 |======================== ext4 soft raid5 4xNVME ........... 5068 |===================================== ext4 Crucial P5 Plus 1TB NVME .... 5137 |====================================== ZFS raidz1 8xNVME ................ 3275 |======================== ZFS raidz1 8xNVME no Compression . 3275 |======================== ZFS mirror 8xNVME ................ 3318 |========================= ext4 soft raid5 8xNVME ........... 5137 |======================================