pts-disk-different-nvmes AMD Ryzen Threadripper 1900X 8-Core testing with a Gigabyte X399 DESIGNARE EX-CF (F13a BIOS) and NVIDIA Quadro P400 on Debian 11 via the Phoronix Test Suite. ZFS raidz1 4xNVME: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1920x1080 ext4 soft raid5 4xNVME: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1920x1080 ext4 Crucial P5 Plus 1TB NVME: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1920x1080 ZFS raidz1 8xNVME: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1920x1080 ZFS raidz1 8xNVME no Compression: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1920x1080 ZFS mirror 8xNVME: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1920x1080 ext4 soft raid5 8xNVME: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1920x1080 ext4 WD_Black SN770 2TB NVMe: Processor: AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads), Motherboard: Gigabyte X399 DESIGNARE EX-CF (F13a BIOS), Chipset: AMD 17h, Memory: 64GB, Disk: Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8, Graphics: NVIDIA Quadro P400, Audio: NVIDIA GP107GL HD Audio, Monitor: DELL S2340T, Network: 4 x Intel I350 + Intel 8265 / 8275 OS: Debian 11, Kernel: 5.10.0-19-amd64 (x86_64), Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1920x1080 SQLite 3.30.1 Threads / Copies: 1 Seconds < Lower Is Better ZFS raidz1 4xNVME ................ 7.852 |=========================== ext4 soft raid5 4xNVME ........... 8.810 |============================== ext4 Crucial P5 Plus 1TB NVME .... 8.367 |============================= ZFS raidz1 8xNVME ................ 8.380 |============================= ZFS raidz1 8xNVME no Compression . 8.379 |============================= ZFS mirror 8xNVME ................ 7.252 |========================= ext4 soft raid5 8xNVME ........... 10.426 |==================================== ext4 WD_Black SN770 2TB NVMe ..... 6.487 |====================== SQLite 3.30.1 Threads / Copies: 8 Seconds < Lower Is Better ZFS raidz1 4xNVME ................ 11.267 |================== ext4 soft raid5 4xNVME ........... 20.005 |================================ ext4 Crucial P5 Plus 1TB NVME .... 15.767 |========================= ZFS raidz1 8xNVME ................ 12.857 |==================== ZFS raidz1 8xNVME no Compression . 12.827 |==================== ZFS mirror 8xNVME ................ 9.043 |============== ext4 soft raid5 8xNVME ........... 22.829 |==================================== ext4 WD_Black SN770 2TB NVMe ..... 21.562 |================================== SQLite 3.30.1 Threads / Copies: 32 Seconds < Lower Is Better ZFS raidz1 4xNVME ................ 23.19 |=================== ext4 soft raid5 4xNVME ........... 31.79 |========================== ext4 Crucial P5 Plus 1TB NVME .... 27.58 |======================= ZFS raidz1 8xNVME ................ 27.00 |====================== ZFS raidz1 8xNVME no Compression . 26.89 |====================== ZFS mirror 8xNVME ................ 17.74 |=============== ext4 soft raid5 8xNVME ........... 35.99 |============================= ext4 WD_Black SN770 2TB NVMe ..... 45.16 |===================================== SQLite 3.30.1 Threads / Copies: 64 Seconds < Lower Is Better ZFS raidz1 4xNVME ................ 37.45 |==================== ext4 soft raid5 4xNVME ........... 45.72 |========================= ext4 Crucial P5 Plus 1TB NVME .... 40.98 |====================== ZFS raidz1 8xNVME ................ 40.30 |====================== ZFS raidz1 8xNVME no Compression . 40.00 |====================== ZFS mirror 8xNVME ................ 36.60 |==================== ext4 soft raid5 8xNVME ........... 50.31 |============================ ext4 WD_Black SN770 2TB NVMe ..... 67.63 |===================================== SQLite 3.30.1 Threads / Copies: 128 Seconds < Lower Is Better ZFS raidz1 4xNVME ................ 76.09 |=============================== ext4 soft raid5 4xNVME ........... 80.15 |================================= ext4 Crucial P5 Plus 1TB NVME .... 59.89 |======================== ZFS raidz1 8xNVME ................ 78.15 |================================ ZFS raidz1 8xNVME no Compression . 77.05 |================================ ZFS mirror 8xNVME ................ 77.88 |================================ ext4 soft raid5 8xNVME ........... 83.86 |================================== ext4 WD_Black SN770 2TB NVMe ..... 90.49 |===================================== Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 4245 |======================== ext4 Crucial P5 Plus 1TB NVME .... 3538 |==================== ZFS raidz1 8xNVME ................ 4037 |======================= ZFS raidz1 8xNVME no Compression . 4170 |======================== ZFS mirror 8xNVME ................ 6617 |====================================== ext4 WD_Black SN770 2TB NVMe ..... 3550 |==================== Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS raidz1 4xNVME ................ 2119 |====== ext4 soft raid5 4xNVME ........... 6666 |=================== ext4 Crucial P5 Plus 1TB NVME .... 1765 |===== ZFS raidz1 8xNVME ................ 2015 |====== ZFS raidz1 8xNVME no Compression . 2081 |====== ZFS mirror 8xNVME ................ 3305 |========= ext4 soft raid5 8xNVME ........... 13225 |===================================== ext4 WD_Black SN770 2TB NVMe ..... 1771 |===== Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 194 |===== ext4 soft raid5 4xNVME ........... 1199 |============================ ext4 Crucial P5 Plus 1TB NVME .... 1286 |============================== ZFS raidz1 8xNVME ................ 224 |===== ZFS raidz1 8xNVME no Compression . 219 |===== ZFS mirror 8xNVME ................ 877 |===================== ext4 soft raid5 8xNVME ........... 1065 |========================= ext4 WD_Black SN770 2TB NVMe ..... 1616 |====================================== Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS raidz1 4xNVME ................ 49567 |==== ext4 soft raid5 4xNVME ........... 306667 |=========================== ext4 Crucial P5 Plus 1TB NVME .... 329667 |============================= ZFS raidz1 8xNVME ................ 57433 |===== ZFS raidz1 8xNVME no Compression . 56167 |===== ZFS mirror 8xNVME ................ 224333 |==================== ext4 soft raid5 8xNVME ........... 272667 |======================== ext4 WD_Black SN770 2TB NVMe ..... 414000 |==================================== Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 2458 |========================== ext4 soft raid5 4xNVME ........... 958 |========== ext4 Crucial P5 Plus 1TB NVME .... 3252 |=================================== ZFS raidz1 8xNVME ................ 2661 |============================ ZFS raidz1 8xNVME no Compression . 2681 |============================= ZFS mirror 8xNVME ................ 3573 |====================================== ext4 soft raid5 8xNVME ........... 1032 |=========== ext4 WD_Black SN770 2TB NVMe ..... 3325 |=================================== Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS raidz1 4xNVME ................ 1225 |========================== ext4 soft raid5 4xNVME ........... 476 |========== ext4 Crucial P5 Plus 1TB NVME .... 1623 |=================================== ZFS raidz1 8xNVME ................ 1327 |============================ ZFS raidz1 8xNVME no Compression . 1337 |============================ ZFS mirror 8xNVME ................ 1783 |====================================== ext4 soft raid5 8xNVME ........... 512 |=========== ext4 WD_Black SN770 2TB NVMe ..... 1659 |=================================== Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 162 |==== ext4 soft raid5 4xNVME ........... 324 |======== ext4 Crucial P5 Plus 1TB NVME .... 878 |====================== ZFS raidz1 8xNVME ................ 177 |===== ZFS raidz1 8xNVME no Compression . 188 |===== ZFS mirror 8xNVME ................ 267 |======= ext4 soft raid5 8xNVME ........... 293 |======= ext4 WD_Black SN770 2TB NVMe ..... 1491 |====================================== Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS raidz1 4xNVME ................ 41500 |==== ext4 soft raid5 4xNVME ........... 82967 |======== ext4 Crucial P5 Plus 1TB NVME .... 224667 |===================== ZFS raidz1 8xNVME ................ 45367 |==== ZFS raidz1 8xNVME no Compression . 48200 |===== ZFS mirror 8xNVME ................ 68333 |====== ext4 soft raid5 8xNVME ........... 75067 |======= ext4 WD_Black SN770 2TB NVMe ..... 381400 |==================================== Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 4075 |===================================== ext4 Crucial P5 Plus 1TB NVME .... 3538 |================================ ZFS raidz1 8xNVME ................ 4005 |===================================== ZFS raidz1 8xNVME no Compression . 3998 |===================================== ZFS mirror 8xNVME ................ 4154 |====================================== ext4 WD_Black SN770 2TB NVMe ..... 3558 |================================= Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS raidz1 4xNVME ................ 2034 |====== ext4 soft raid5 4xNVME ........... 6706 |=================== ext4 Crucial P5 Plus 1TB NVME .... 1765 |===== ZFS raidz1 8xNVME ................ 1999 |====== ZFS raidz1 8xNVME no Compression . 1995 |====== ZFS mirror 8xNVME ................ 2073 |====== ext4 soft raid5 8xNVME ........... 13200 |===================================== ext4 WD_Black SN770 2TB NVMe ..... 1775 |===== Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 1287 |===================================== ext4 soft raid5 4xNVME ........... 865 |========================= ext4 Crucial P5 Plus 1TB NVME .... 1311 |====================================== ZFS raidz1 8xNVME ................ 1275 |===================================== ZFS raidz1 8xNVME no Compression . 1294 |====================================== ZFS mirror 8xNVME ................ 1296 |====================================== ext4 soft raid5 8xNVME ........... 779 |======================= ext4 WD_Black SN770 2TB NVMe ..... 357 |========== Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS raidz1 4xNVME ................ 329333 |=================================== ext4 soft raid5 4xNVME ........... 221333 |======================== ext4 Crucial P5 Plus 1TB NVME .... 335667 |==================================== ZFS raidz1 8xNVME ................ 326333 |=================================== ZFS raidz1 8xNVME no Compression . 331000 |=================================== ZFS mirror 8xNVME ................ 332000 |==================================== ext4 soft raid5 8xNVME ........... 199667 |===================== ext4 WD_Black SN770 2TB NVMe ..... 91333 |========== Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 2065 |====================== ext4 soft raid5 4xNVME ........... 1308 |============== ext4 Crucial P5 Plus 1TB NVME .... 3236 |=================================== ZFS raidz1 8xNVME ................ 2381 |========================= ZFS raidz1 8xNVME no Compression . 2446 |========================== ZFS mirror 8xNVME ................ 3555 |====================================== ext4 soft raid5 8xNVME ........... 1636 |================= ext4 WD_Black SN770 2TB NVMe ..... 3345 |==================================== Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS raidz1 4xNVME ................ 1029 |====================== ext4 soft raid5 4xNVME ........... 650 |============== ext4 Crucial P5 Plus 1TB NVME .... 1614 |=================================== ZFS raidz1 8xNVME ................ 1186 |========================= ZFS raidz1 8xNVME no Compression . 1220 |========================== ZFS mirror 8xNVME ................ 1774 |====================================== ext4 soft raid5 8xNVME ........... 814 |================= ext4 WD_Black SN770 2TB NVMe ..... 1669 |==================================== Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 670 |================= ext4 soft raid5 4xNVME ........... 449 |=========== ext4 Crucial P5 Plus 1TB NVME .... 891 |====================== ZFS raidz1 8xNVME ................ 665 |================= ZFS raidz1 8xNVME no Compression . 682 |================= ZFS mirror 8xNVME ................ 771 |=================== ext4 soft raid5 8xNVME ........... 387 |========== ext4 WD_Black SN770 2TB NVMe ..... 1514 |====================================== Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better ZFS raidz1 4xNVME ................ 171333 |================ ext4 soft raid5 4xNVME ........... 115000 |=========== ext4 Crucial P5 Plus 1TB NVME .... 228000 |===================== ZFS raidz1 8xNVME ................ 170667 |================ ZFS raidz1 8xNVME no Compression . 174667 |================ ZFS mirror 8xNVME ................ 197333 |================== ext4 soft raid5 8xNVME ........... 99067 |========= ext4 WD_Black SN770 2TB NVMe ..... 387333 |==================================== FS-Mark 3.3 Test: 1000 Files, 1MB Size Files/s > Higher Is Better ZFS raidz1 4xNVME ................ 646.5 |================================== ext4 soft raid5 4xNVME ........... 254.0 |============== ext4 Crucial P5 Plus 1TB NVME .... 584.1 |=============================== ZFS raidz1 8xNVME ................ 646.7 |================================== ZFS raidz1 8xNVME no Compression . 589.7 |=============================== ZFS mirror 8xNVME ................ 581.1 |=============================== ext4 soft raid5 8xNVME ........... 264.4 |============== ext4 WD_Black SN770 2TB NVMe ..... 693.8 |===================================== FS-Mark 3.3 Test: 5000 Files, 1MB Size, 4 Threads Files/s > Higher Is Better ZFS raidz1 4xNVME ................ 1632.7 |================================== ext4 soft raid5 4xNVME ........... 537.9 |=========== ext4 Crucial P5 Plus 1TB NVME .... 522.1 |=========== ZFS raidz1 8xNVME ................ 1655.6 |=================================== ZFS raidz1 8xNVME no Compression . 1240.0 |========================== ZFS mirror 8xNVME ................ 1711.4 |==================================== ext4 soft raid5 8xNVME ........... 570.0 |============ ext4 WD_Black SN770 2TB NVMe ..... 1631.5 |================================== FS-Mark 3.3 Test: 4000 Files, 32 Sub Dirs, 1MB Size Files/s > Higher Is Better ZFS raidz1 4xNVME ................ 684.2 |===================================== ext4 soft raid5 4xNVME ........... 284.7 |=============== ext4 Crucial P5 Plus 1TB NVME .... 435.0 |======================== ZFS raidz1 8xNVME ................ 644.7 |=================================== ZFS raidz1 8xNVME no Compression . 592.1 |================================ ZFS mirror 8xNVME ................ 658.2 |==================================== ext4 soft raid5 8xNVME ........... 286.6 |=============== ext4 WD_Black SN770 2TB NVMe ..... 558.2 |============================== FS-Mark 3.3 Test: 1000 Files, 1MB Size, No Sync/FSync Files/s > Higher Is Better ZFS raidz1 4xNVME ................ 1359.3 |=========================== ext4 soft raid5 4xNVME ........... 1733.3 |=================================== ext4 Crucial P5 Plus 1TB NVME .... 1791.6 |==================================== ZFS raidz1 8xNVME ................ 1340.7 |=========================== ZFS raidz1 8xNVME no Compression . 1160.9 |======================= ZFS mirror 8xNVME ................ 1338.4 |=========================== ext4 soft raid5 8xNVME ........... 1782.9 |==================================== ext4 WD_Black SN770 2TB NVMe ..... 1806.6 |==================================== Dbench 4.0 12 Clients MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 2626.03 |============================= ext4 soft raid5 4xNVME ........... 2484.85 |=========================== ext4 Crucial P5 Plus 1TB NVME .... 658.79 |======= ZFS raidz1 8xNVME ................ 2472.42 |=========================== ZFS raidz1 8xNVME no Compression . 2783.64 |============================== ZFS mirror 8xNVME ................ 2730.75 |============================== ext4 soft raid5 8xNVME ........... 2330.16 |========================== ext4 WD_Black SN770 2TB NVMe ..... 3196.68 |=================================== Dbench 4.0 1 Clients MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 410.17 |========================= ext4 soft raid5 4xNVME ........... 453.17 |============================ ext4 Crucial P5 Plus 1TB NVME .... 102.99 |====== ZFS raidz1 8xNVME ................ 380.35 |======================== ZFS raidz1 8xNVME no Compression . 458.91 |============================= ZFS mirror 8xNVME ................ 435.70 |=========================== ext4 soft raid5 8xNVME ........... 423.83 |========================== ext4 WD_Black SN770 2TB NVMe ..... 579.24 |==================================== Compile Bench 0.6 Test: Compile MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 1398.57 |================================ ext4 soft raid5 4xNVME ........... 1478.00 |================================== ext4 Crucial P5 Plus 1TB NVME .... 1483.07 |================================== ZFS raidz1 8xNVME ................ 1390.71 |================================ ZFS raidz1 8xNVME no Compression . 1247.36 |============================= ZFS mirror 8xNVME ................ 1408.42 |================================= ext4 soft raid5 8xNVME ........... 1474.80 |================================== ext4 WD_Black SN770 2TB NVMe ..... 1513.97 |=================================== Compile Bench 0.6 Test: Initial Create MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 222.04 |=================== ext4 soft raid5 4xNVME ........... 411.07 |=================================== ext4 Crucial P5 Plus 1TB NVME .... 423.67 |==================================== ZFS raidz1 8xNVME ................ 221.83 |=================== ZFS raidz1 8xNVME no Compression . 206.77 |================== ZFS mirror 8xNVME ................ 216.21 |================== ext4 soft raid5 8xNVME ........... 413.46 |=================================== ext4 WD_Black SN770 2TB NVMe ..... 420.47 |==================================== Compile Bench 0.6 Test: Read Compiled Tree MB/s > Higher Is Better ZFS raidz1 4xNVME ................ 1212.73 |=============== ext4 soft raid5 4xNVME ........... 2642.03 |================================== ext4 Crucial P5 Plus 1TB NVME .... 2747.01 |=================================== ZFS raidz1 8xNVME ................ 1217.80 |================ ZFS raidz1 8xNVME no Compression . 1694.75 |====================== ZFS mirror 8xNVME ................ 1454.31 |=================== ext4 soft raid5 8xNVME ........... 2683.76 |================================== ext4 WD_Black SN770 2TB NVMe ..... 2738.07 |=================================== PostMark 1.51 Disk Transaction Performance TPS > Higher Is Better ZFS raidz1 4xNVME ................ 3289 |======================== ext4 soft raid5 4xNVME ........... 5068 |===================================== ext4 Crucial P5 Plus 1TB NVME .... 5137 |===================================== ZFS raidz1 8xNVME ................ 3275 |======================== ZFS raidz1 8xNVME no Compression . 3275 |======================== ZFS mirror 8xNVME ................ 3318 |======================== ext4 soft raid5 8xNVME ........... 5137 |===================================== ext4 WD_Black SN770 2TB NVMe ..... 5211 |======================================