45drivesxl60 2 x AMD EPYC 7513 32-Core testing with a Dell PowerEdge R7525 [0590KW] (2.6.6 BIOS) and Matrox G200eW3 on Debian GNU/Linux 11 via the Phoronix Test Suite. 10 Per VDEV RaidZ2: Processor: Intel Xeon Silver 4216 (32 Cores), Motherboard: 45Drives XL60 0123456789, Memory: 256GB, Disk: 742117GB OS: FreeBSD, Kernel: 12.2-RELEASE-p10 (x86_64), Compiler: Clang 10.0.1, File-System: zfs 5 Per VDEV RaidZ2: Processor: Intel Xeon Silver 4216 (32 Cores), Motherboard: 45Drives XL60 0123456789, Memory: 256GB, Disk: 742117GB OS: FreeBSD, Kernel: 12.2-RELEASE-p10 (x86_64), Compiler: Clang 10.0.1, File-System: zfs Mirrors: Processor: Intel Xeon Silver 4216 (32 Cores), Motherboard: 45Drives XL60 0123456789, Memory: 256GB, Disk: 742117GB OS: FreeBSD, Kernel: 12.2-RELEASE-p10 (x86_64), Compiler: Clang 10.0.1, File-System: zfs DELL ENT NVME SSD1: Processor: 2 x AMD EPYC 7513 32-Core (64 Cores / 128 Threads), Motherboard: Dell PowerEdge R7525 [0590KW] (2.6.6 BIOS), Chipset: AMD Starship/Matisse, Memory: 8 x 64 GB DDR4-3200MT/s 36ASF8G72PZ-3G2B2, Disk: 240GB DELLBOSS VD + 2 x 1920GB Dell Ent NVMe CM + 2 x 3841GB SAMSUNG MZQL23T8, Graphics: Matrox G200eW3, Monitor: DELL E172FP, Network: 4 x Intel X710 for 10GBASE-T + 2 x Broadcom NetXtreme BCM5720 2-port PCIe OS: Debian GNU/Linux 11, Kernel: 5.15.30-2-pve (x86_64), Vulkan: 1.0.2, Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1280x1024 DELL SAMSUNG SSD4: Processor: 2 x AMD EPYC 7513 32-Core (64 Cores / 128 Threads), Motherboard: Dell PowerEdge R7525 [0590KW] (2.6.6 BIOS), Chipset: AMD Starship/Matisse, Memory: 8 x 64 GB DDR4-3200MT/s 36ASF8G72PZ-3G2B2, Disk: 240GB DELLBOSS VD + 2 x 1920GB Dell Ent NVMe CM + 2 x 3841GB SAMSUNG MZQL23T8, Graphics: Matrox G200eW3, Monitor: DELL E172FP, Network: 4 x Intel X710 for 10GBASE-T + 2 x Broadcom NetXtreme BCM5720 2-port PCIe OS: Debian GNU/Linux 11, Kernel: 5.15.30-2-pve (x86_64), Vulkan: 1.0.2, Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1280x1024 DELL ENTERPRISE NVME: Processor: 2 x AMD EPYC 7513 32-Core (64 Cores / 128 Threads), Motherboard: Dell PowerEdge R7525 [0590KW] (2.6.6 BIOS), Chipset: AMD Starship/Matisse, Memory: 8 x 64 GB DDR4-3200MT/s 36ASF8G72PZ-3G2B2, Disk: 240GB DELLBOSS VD + 2 x 1920GB Dell Ent NVMe CM + 2 x 3841GB SAMSUNG MZQL23T8, Graphics: Matrox G200eW3, Monitor: DELL E172FP, Network: 4 x Intel X710 for 10GBASE-T + 2 x Broadcom NetXtreme BCM5720 2-port PCIe OS: Debian GNU/Linux 11, Kernel: 5.15.30-2-pve (x86_64), Vulkan: 1.0.2, Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1280x1024 SAMSUNG ZFS MIRROR: Processor: 2 x AMD EPYC 7513 32-Core (64 Cores / 128 Threads), Motherboard: Dell PowerEdge R7525 [0590KW] (2.6.6 BIOS), Chipset: AMD Starship/Matisse, Memory: 8 x 64 GB DDR4-3200MT/s 36ASF8G72PZ-3G2B2, Disk: 240GB DELLBOSS VD + 2 x 1920GB Dell Ent NVMe CM + 2 x 3841GB SAMSUNG MZQL23T8, Graphics: Matrox G200eW3, Monitor: DELL E172FP, Network: 4 x Intel X710 for 10GBASE-T + 2 x Broadcom NetXtreme BCM5720 2-port PCIe OS: Debian GNU/Linux 11, Kernel: 5.15.30-2-pve (x86_64), Vulkan: 1.0.2, Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1280x1024 DELLENT ZFS MIRROR: Processor: 2 x AMD EPYC 7513 32-Core (64 Cores / 128 Threads), Motherboard: Dell PowerEdge R7525 [0590KW] (2.6.6 BIOS), Chipset: AMD Starship/Matisse, Memory: 8 x 64 GB DDR4-3200MT/s 36ASF8G72PZ-3G2B2, Disk: 240GB DELLBOSS VD + 2 x 1920GB Dell Ent NVMe CM + 2 x 3841GB SAMSUNG MZQL23T8, Graphics: Matrox G200eW3, Monitor: DELL E172FP, Network: 4 x Intel X710 for 10GBASE-T + 2 x Broadcom NetXtreme BCM5720 2-port PCIe OS: Debian GNU/Linux 11, Kernel: 5.15.30-2-pve (x86_64), Vulkan: 1.0.2, Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1280x1024 PERC RAID1 SAMSUNG 256-A-D-C: Processor: 2 x AMD EPYC 7513 32-Core (64 Cores / 128 Threads), Motherboard: Dell PowerEdge R7525 [0590KW] (2.6.6 BIOS), Chipset: AMD Starship/Matisse, Memory: 8 x 64 GB DDR4-3200MT/s 36ASF8G72PZ-3G2B2, Disk: 240GB DELLBOSS VD + 2 x 1920GB Dell Ent NVMe CM + 2 x 3841GB SAMSUNG MZQL23T8, Graphics: Matrox G200eW3, Monitor: DELL E172FP, Network: 4 x Intel X710 for 10GBASE-T + 2 x Broadcom NetXtreme BCM5720 2-port PCIe OS: Debian GNU/Linux 11, Kernel: 5.15.30-2-pve (x86_64), Vulkan: 1.0.2, Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1280x1024 PERC RAID1 DELL ENT. H755: Processor: 2 x AMD EPYC 7513 32-Core (64 Cores / 128 Threads), Motherboard: Dell PowerEdge R7525 [0590KW] (2.6.6 BIOS), Chipset: AMD Starship/Matisse, Memory: 8 x 64 GB DDR4-3200MT/s 36ASF8G72PZ-3G2B2, Disk: 240GB DELLBOSS VD + 2 x 1920GB Dell Ent NVMe CM + 2 x 3841GB SAMSUNG MZQL23T8, Graphics: Matrox G200eW3, Monitor: DELL E172FP, Network: 4 x Intel X710 for 10GBASE-T + 2 x Broadcom NetXtreme BCM5720 2-port PCIe OS: Debian GNU/Linux 11, Kernel: 5.15.30-2-pve (x86_64), Vulkan: 1.0.2, Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1280x1024 NFS 1 NO SSD: Processor: 2 x AMD EPYC 7513 32-Core (64 Cores / 128 Threads), Motherboard: Dell PowerEdge R7525 [0590KW] (2.6.6 BIOS), Chipset: AMD Starship/Matisse, Memory: 8 x 64 GB DDR4-3200MT/s 36ASF8G72PZ-3G2B2, Disk: 240GB DELLBOSS VD + 2 x 1920GB Dell Ent NVMe CM + 2 x 3841GB SAMSUNG MZQL23T8, Graphics: Matrox G200eW3, Monitor: DELL E172FP, Network: 4 x Intel X710 for 10GBASE-T + 2 x Broadcom NetXtreme BCM5720 2-port PCIe OS: Debian GNU/Linux 11, Kernel: 5.15.30-2-pve (x86_64), Vulkan: 1.0.2, Compiler: GCC 10.2.1 20210110, File-System: nfs, Screen Resolution: 1280x1024 Flexible IO Tester 3.25 Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory MB/s > Higher Is Better 10 Per VDEV RaidZ2 ........... 9131 |========================================== DELL ENT NVME SSD1 ........... 203 |= DELL SAMSUNG SSD4 ............ 2080 |========== DELL ENTERPRISE NVME ......... 2285 |=========== SAMSUNG ZFS MIRROR ........... 6145 |============================ DELLENT ZFS MIRROR ........... 6074 |============================ PERC RAID1 SAMSUNG 256-A-D-C . 2801 |============= PERC RAID1 DELL ENT. H755 .... 2475 |=========== NFS 1 NO SSD ................. 194 |= Flexible IO Tester 3.25 Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better 10 Per VDEV RaidZ2 ........... 92300 |==================== 5 Per VDEV RaidZ2 ............ 110333 |======================== Mirrors ...................... 129333 |============================ DELL ENT NVME SSD1 ........... 18260 |==== DELL SAMSUNG SSD4 ............ 6242 |= DELL ENTERPRISE NVME ......... 61233 |============= SAMSUNG ZFS MIRROR ........... 180000 |======================================= DELLENT ZFS MIRROR ........... 185000 |======================================== PERC RAID1 SAMSUNG 256-A-D-C . 32633 |======= PERC RAID1 DELL ENT. H755 .... 42942 |========= NFS 1 NO SSD ................. 2215 | Flexible IO Tester 3.25 Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better 10 Per VDEV RaidZ2 ........... 360.000 |=================== 5 Per VDEV RaidZ2 ............ 432.000 |======================= Mirrors ...................... 505.000 |=========================== DELL ENT NVME SSD1 ........... 71.300 |==== DELL SAMSUNG SSD4 ............ 24.400 |= DELL ENTERPRISE NVME ......... 240.000 |============= SAMSUNG ZFS MIRROR ........... 703.000 |====================================== DELLENT ZFS MIRROR ........... 721.000 |======================================= PERC RAID1 SAMSUNG 256-A-D-C . 128.000 |======= PERC RAID1 DELL ENT. H755 .... 168.000 |========= NFS 1 NO SSD ................. 8.875 | Flexible IO Tester 3.25 Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better 10 Per VDEV RaidZ2 ........... 127000 |============== 5 Per VDEV RaidZ2 ............ 135667 |=============== Mirrors ...................... 137667 |================ DELL ENT NVME SSD1 ........... 17871 |== DELL SAMSUNG SSD4 ............ 6385 |= DELL ENTERPRISE NVME ......... 61500 |======= SAMSUNG ZFS MIRROR ........... 353917 |======================================== DELLENT ZFS MIRROR ........... 333333 |====================================== PERC RAID1 SAMSUNG 256-A-D-C . 39767 |==== PERC RAID1 DELL ENT. H755 .... 48167 |===== NFS 1 NO SSD ................. 2319 | Flexible IO Tester 3.25 Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better 10 Per VDEV RaidZ2 ........... 495.000 |============== 5 Per VDEV RaidZ2 ............ 529.000 |=============== Mirrors ...................... 537.000 |=============== DELL ENT NVME SSD1 ........... 70.300 |== DELL SAMSUNG SSD4 ............ 24.300 |= DELL ENTERPRISE NVME ......... 240.000 |======= SAMSUNG ZFS MIRROR ........... 1382.000 |====================================== DELLENT ZFS MIRROR ........... 1302.000 |==================================== PERC RAID1 SAMSUNG 256-A-D-C . 157.000 |==== PERC RAID1 DELL ENT. H755 .... 188.000 |===== NFS 1 NO SSD ................. 9.289 | Flexible IO Tester 3.25 Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory IOPS > Higher Is Better 10 Per VDEV RaidZ2 ........... 73033 |============================== 5 Per VDEV RaidZ2 ............ 91300 |===================================== Mirrors ...................... 99900 |========================================= DELL ENT NVME SSD1 ........... 1623 |= DELL SAMSUNG SSD4 ............ 16633 |======= DELL ENTERPRISE NVME ......... 18300 |======== SAMSUNG ZFS MIRROR ........... 49140 |==================== DELLENT ZFS MIRROR ........... 48446 |==================== PERC RAID1 SAMSUNG 256-A-D-C . 22400 |========= PERC RAID1 DELL ENT. H755 .... 19787 |======== NFS 1 NO SSD ................. 1544 |= Flexible IO Tester 3.25 Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory IOPS > Higher Is Better 10 Per VDEV RaidZ2 ........... 146933 |======================================== 5 Per VDEV RaidZ2 ............ 131333 |==================================== Mirrors ...................... 132600 |==================================== DELL ENT NVME SSD1 ........... 1553 | DELL SAMSUNG SSD4 ............ 17133 |===== DELL ENTERPRISE NVME ......... 9457 |=== SAMSUNG ZFS MIRROR ........... 54433 |=============== DELLENT ZFS MIRROR ........... 52847 |============== PERC RAID1 SAMSUNG 256-A-D-C . 21700 |====== PERC RAID1 DELL ENT. H755 .... 13167 |==== NFS 1 NO SSD ................. 2166 |= Flexible IO Tester 3.25 Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better 10 Per VDEV RaidZ2 ........... 188000 |================= 5 Per VDEV RaidZ2 ............ 188000 |================= Mirrors ...................... 189333 |================= DELL ENT NVME SSD1 ........... 7183 |= DELL SAMSUNG SSD4 ............ 63067 |====== DELL ENTERPRISE NVME ......... 22500 |== SAMSUNG ZFS MIRROR ........... 440133 |======================================== DELLENT ZFS MIRROR ........... 434467 |======================================= PERC RAID1 SAMSUNG 256-A-D-C . 34367 |=== PERC RAID1 DELL ENT. H755 .... 52100 |===== NFS 1 NO SSD ................. 2488 | Flexible IO Tester 3.25 Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better 10 Per VDEV RaidZ2 ........... 735.000 |================ 5 Per VDEV RaidZ2 ............ 735.000 |================ Mirrors ...................... 739.000 |================ DELL ENT NVME SSD1 ........... 28.100 |= DELL SAMSUNG SSD4 ............ 246.000 |===== DELL ENTERPRISE NVME ......... 87.600 |== SAMSUNG ZFS MIRROR ........... 1720.000 |====================================== DELLENT ZFS MIRROR ........... 1698.000 |====================================== PERC RAID1 SAMSUNG 256-A-D-C . 134.000 |=== PERC RAID1 DELL ENT. H755 .... 204.000 |===== NFS 1 NO SSD ................. 9.967 | Flexible IO Tester 3.25 Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better 10 Per VDEV RaidZ2 ........... 685.000 |====================================== 5 Per VDEV RaidZ2 ............ 692.000 |======================================= Mirrors ...................... 697.000 |======================================= DELL ENT NVME SSD1 ........... 27.300 |== DELL SAMSUNG SSD4 ............ 38.400 |== DELL ENTERPRISE NVME ......... 47.200 |=== SAMSUNG ZFS MIRROR ........... 388.000 |====================== DELLENT ZFS MIRROR ........... 386.000 |====================== PERC RAID1 SAMSUNG 256-A-D-C . 80.500 |===== PERC RAID1 DELL ENT. H755 .... 183.000 |========== NFS 1 NO SSD ................. 9.904 |= Flexible IO Tester 3.25 Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better Mirrors ...................... 178333 |======================================== 10 Per VDEV RaidZ2 ........... 175333 |======================================= 5 Per VDEV RaidZ2 ............ 177333 |======================================== Mirrors ...................... 178667 |======================================== DELL ENT NVME SSD1 ........... 6991 |== DELL SAMSUNG SSD4 ............ 9825 |== DELL ENTERPRISE NVME ......... 12100 |=== SAMSUNG ZFS MIRROR ........... 99133 |====================== DELLENT ZFS MIRROR ........... 98583 |====================== PERC RAID1 SAMSUNG 256-A-D-C . 20533 |===== PERC RAID1 DELL ENT. H755 .... 46800 |========== NFS 1 NO SSD ................. 2472 |= Flexible IO Tester 3.25 Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory MB/s > Higher Is Better DELL ENT NVME SSD1 ........... 195 |= DELL SAMSUNG SSD4 ............ 2138 |============= DELL ENTERPRISE NVME ......... 1182 |======= SAMSUNG ZFS MIRROR ........... 6804 |========================================== DELLENT ZFS MIRROR ........... 6622 |========================================= PERC RAID1 SAMSUNG 256-A-D-C . 2754 |================= PERC RAID1 DELL ENT. H755 .... 1645 |========== NFS 1 NO SSD ................. 271 |== Flexible IO Tester 3.25 Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory IOPS > Higher Is Better 10 Per VDEV RaidZ2 ........... 156667 |======================================== 5 Per VDEV RaidZ2 ............ 157000 |======================================== Mirrors ...................... 156333 |======================================== DELL ENT NVME SSD1 ........... 1561 | DELL SAMSUNG SSD4 ............ 6327 |== DELL ENTERPRISE NVME ......... 7633 |== SAMSUNG ZFS MIRROR ........... 58733 |=============== DELLENT ZFS MIRROR ........... 57367 |=============== PERC RAID1 SAMSUNG 256-A-D-C . 9613 |== PERC RAID1 DELL ENT. H755 .... 12767 |=== NFS 1 NO SSD ................. 2139 |= Flexible IO Tester 3.25 Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory MB/s > Higher Is Better DELL ENT NVME SSD1 ........... 196 |= DELL SAMSUNG SSD4 ............ 791 |===== DELL ENTERPRISE NVME ......... 955 |===== SAMSUNG ZFS MIRROR ........... 7340 |========================================== DELLENT ZFS MIRROR ........... 7172 |========================================= PERC RAID1 SAMSUNG 256-A-D-C . 1202 |======= PERC RAID1 DELL ENT. H755 .... 1591 |========= NFS 1 NO SSD ................. 268 |== Flexible IO Tester 3.25 Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory IOPS > Higher Is Better 10 Per VDEV RaidZ2 ........... 72700 |=============================== 5 Per VDEV RaidZ2 ............ 93500 |======================================== Mirrors ...................... 95733 |========================================= DELL ENT NVME SSD1 ........... 1617 |= DELL SAMSUNG SSD4 ............ 16400 |======= DELL ENTERPRISE NVME ......... 18300 |======== SAMSUNG ZFS MIRROR ........... 49633 |===================== DELLENT ZFS MIRROR ........... 51633 |====================== PERC RAID1 SAMSUNG 256-A-D-C . 22400 |========== PERC RAID1 DELL ENT. H755 .... 18700 |======== NFS 1 NO SSD ................. 1534 |= Flexible IO Tester 3.25 Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory MB/s > Higher Is Better 10 Per VDEV RaidZ2 ........... 9083 |========================================== DELL ENT NVME SSD1 ........... 203 |= DELL SAMSUNG SSD4 ............ 2054 |========= DELL ENTERPRISE NVME ......... 2286 |=========== SAMSUNG ZFS MIRROR ........... 6210 |============================= DELLENT ZFS MIRROR ........... 6454 |============================== PERC RAID1 SAMSUNG 256-A-D-C . 2798 |============= PERC RAID1 DELL ENT. H755 .... 2334 |=========== NFS 1 NO SSD ................. 192 |=