45drivesxl60 2 x AMD EPYC 7513 32-Core testing with a Dell PowerEdge R7525 [0590KW] (2.6.6 BIOS) and Matrox G200eW3 on Debian GNU/Linux 11 via the Phoronix Test Suite. 10 Per VDEV RaidZ2: Processor: Intel Xeon Silver 4216 (32 Cores), Motherboard: 45Drives XL60 0123456789, Memory: 256GB, Disk: 742117GB OS: FreeBSD, Kernel: 12.2-RELEASE-p10 (x86_64), Compiler: Clang 10.0.1, File-System: zfs 5 Per VDEV RaidZ2: Processor: Intel Xeon Silver 4216 (32 Cores), Motherboard: 45Drives XL60 0123456789, Memory: 256GB, Disk: 742117GB OS: FreeBSD, Kernel: 12.2-RELEASE-p10 (x86_64), Compiler: Clang 10.0.1, File-System: zfs Mirrors: Processor: Intel Xeon Silver 4216 (32 Cores), Motherboard: 45Drives XL60 0123456789, Memory: 256GB, Disk: 742117GB OS: FreeBSD, Kernel: 12.2-RELEASE-p10 (x86_64), Compiler: Clang 10.0.1, File-System: zfs DELL ENT NVME SSD1: Processor: 2 x AMD EPYC 7513 32-Core (64 Cores / 128 Threads), Motherboard: Dell PowerEdge R7525 [0590KW] (2.6.6 BIOS), Chipset: AMD Starship/Matisse, Memory: 8 x 64 GB DDR4-3200MT/s 36ASF8G72PZ-3G2B2, Disk: 240GB DELLBOSS VD + 2 x 1920GB Dell Ent NVMe CM + 2 x 3841GB SAMSUNG MZQL23T8, Graphics: Matrox G200eW3, Monitor: DELL E172FP, Network: 4 x Intel X710 for 10GBASE-T + 2 x Broadcom NetXtreme BCM5720 2-port PCIe OS: Debian GNU/Linux 11, Kernel: 5.15.30-2-pve (x86_64), Vulkan: 1.0.2, Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1280x1024 DELL SAMSUNG SSD4: Processor: 2 x AMD EPYC 7513 32-Core (64 Cores / 128 Threads), Motherboard: Dell PowerEdge R7525 [0590KW] (2.6.6 BIOS), Chipset: AMD Starship/Matisse, Memory: 8 x 64 GB DDR4-3200MT/s 36ASF8G72PZ-3G2B2, Disk: 240GB DELLBOSS VD + 2 x 1920GB Dell Ent NVMe CM + 2 x 3841GB SAMSUNG MZQL23T8, Graphics: Matrox G200eW3, Monitor: DELL E172FP, Network: 4 x Intel X710 for 10GBASE-T + 2 x Broadcom NetXtreme BCM5720 2-port PCIe OS: Debian GNU/Linux 11, Kernel: 5.15.30-2-pve (x86_64), Vulkan: 1.0.2, Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1280x1024 DELL ENTERPRISE NVME: Processor: 2 x AMD EPYC 7513 32-Core (64 Cores / 128 Threads), Motherboard: Dell PowerEdge R7525 [0590KW] (2.6.6 BIOS), Chipset: AMD Starship/Matisse, Memory: 8 x 64 GB DDR4-3200MT/s 36ASF8G72PZ-3G2B2, Disk: 240GB DELLBOSS VD + 2 x 1920GB Dell Ent NVMe CM + 2 x 3841GB SAMSUNG MZQL23T8, Graphics: Matrox G200eW3, Monitor: DELL E172FP, Network: 4 x Intel X710 for 10GBASE-T + 2 x Broadcom NetXtreme BCM5720 2-port PCIe OS: Debian GNU/Linux 11, Kernel: 5.15.30-2-pve (x86_64), Vulkan: 1.0.2, Compiler: GCC 10.2.1 20210110, File-System: ext4, Screen Resolution: 1280x1024 SAMSUNG ZFS MIRROR: Processor: 2 x AMD EPYC 7513 32-Core (64 Cores / 128 Threads), Motherboard: Dell PowerEdge R7525 [0590KW] (2.6.6 BIOS), Chipset: AMD Starship/Matisse, Memory: 8 x 64 GB DDR4-3200MT/s 36ASF8G72PZ-3G2B2, Disk: 240GB DELLBOSS VD + 2 x 1920GB Dell Ent NVMe CM + 2 x 3841GB SAMSUNG MZQL23T8, Graphics: Matrox G200eW3, Monitor: DELL E172FP, Network: 4 x Intel X710 for 10GBASE-T + 2 x Broadcom NetXtreme BCM5720 2-port PCIe OS: Debian GNU/Linux 11, Kernel: 5.15.30-2-pve (x86_64), Vulkan: 1.0.2, Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1280x1024 Flexible IO Tester 3.25 Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better 10 Per VDEV RaidZ2 ... 685.0 |================================================ 5 Per VDEV RaidZ2 .... 692.0 |================================================= Mirrors .............. 697.0 |================================================= DELL ENT NVME SSD1 ... 27.3 |== DELL SAMSUNG SSD4 .... 38.4 |=== DELL ENTERPRISE NVME . 47.2 |=== SAMSUNG ZFS MIRROR ... 388.0 |=========================== Flexible IO Tester 3.25 Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better Mirrors .............. 178333 |================================================ 10 Per VDEV RaidZ2 ... 175333 |=============================================== 5 Per VDEV RaidZ2 .... 177333 |================================================ Mirrors .............. 178667 |================================================ DELL ENT NVME SSD1 ... 6991 |== DELL SAMSUNG SSD4 .... 9825 |=== DELL ENTERPRISE NVME . 12100 |=== SAMSUNG ZFS MIRROR ... 99133 |=========================== Flexible IO Tester 3.25 Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better 10 Per VDEV RaidZ2 ... 360.0 |========================= 5 Per VDEV RaidZ2 .... 432.0 |============================== Mirrors .............. 505.0 |=================================== DELL ENT NVME SSD1 ... 71.3 |===== DELL SAMSUNG SSD4 .... 24.4 |== DELL ENTERPRISE NVME . 240.0 |================= SAMSUNG ZFS MIRROR ... 703.0 |================================================= Flexible IO Tester 3.25 Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better 10 Per VDEV RaidZ2 ... 92300 |========================= 5 Per VDEV RaidZ2 .... 110333 |============================= Mirrors .............. 129333 |================================== DELL ENT NVME SSD1 ... 18260 |===== DELL SAMSUNG SSD4 .... 6242 |== DELL ENTERPRISE NVME . 61233 |================ SAMSUNG ZFS MIRROR ... 180000 |================================================ Flexible IO Tester 3.25 Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory IOPS > Higher Is Better 10 Per VDEV RaidZ2 ... 156667 |================================================ 5 Per VDEV RaidZ2 .... 157000 |================================================ Mirrors .............. 156333 |================================================ DELL ENT NVME SSD1 ... 1561 | DELL SAMSUNG SSD4 .... 6327 |== DELL ENTERPRISE NVME . 7633 |== SAMSUNG ZFS MIRROR ... 58733 |================== Flexible IO Tester 3.25 Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory MB/s > Higher Is Better 10 Per VDEV RaidZ2 ... 9083 |================================================== DELL ENT NVME SSD1 ... 203 |= DELL SAMSUNG SSD4 .... 2054 |=========== DELL ENTERPRISE NVME . 2286 |============= SAMSUNG ZFS MIRROR ... 6210 |================================== Flexible IO Tester 3.25 Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory IOPS > Higher Is Better 10 Per VDEV RaidZ2 ... 72700 |===================================== 5 Per VDEV RaidZ2 .... 93500 |================================================ Mirrors .............. 95733 |================================================= DELL ENT NVME SSD1 ... 1617 |= DELL SAMSUNG SSD4 .... 16400 |======== DELL ENTERPRISE NVME . 18300 |========= SAMSUNG ZFS MIRROR ... 49633 |========================= Flexible IO Tester 3.25 Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better 10 Per VDEV RaidZ2 ... 735.0 |===================== 5 Per VDEV RaidZ2 .... 735.0 |===================== Mirrors .............. 739.0 |===================== DELL ENT NVME SSD1 ... 28.1 |= DELL SAMSUNG SSD4 .... 246.0 |======= DELL ENTERPRISE NVME . 87.6 |== SAMSUNG ZFS MIRROR ... 1720.0 |================================================ Flexible IO Tester 3.25 Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better 10 Per VDEV RaidZ2 ... 188000 |===================== 5 Per VDEV RaidZ2 .... 188000 |===================== Mirrors .............. 189333 |===================== DELL ENT NVME SSD1 ... 7183 |= DELL SAMSUNG SSD4 .... 63067 |======= DELL ENTERPRISE NVME . 22500 |== SAMSUNG ZFS MIRROR ... 440133 |================================================ Flexible IO Tester 3.25 Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better 10 Per VDEV RaidZ2 ... 495.0 |================= 5 Per VDEV RaidZ2 .... 529.0 |================== Mirrors .............. 537.0 |=================== DELL ENT NVME SSD1 ... 70.3 |== DELL SAMSUNG SSD4 .... 24.3 |= DELL ENTERPRISE NVME . 240.0 |======== SAMSUNG ZFS MIRROR ... 1382.0 |================================================ Flexible IO Tester 3.25 Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better 10 Per VDEV RaidZ2 ... 127000 |================= 5 Per VDEV RaidZ2 .... 135667 |================== Mirrors .............. 137667 |=================== DELL ENT NVME SSD1 ... 17871 |== DELL SAMSUNG SSD4 .... 6385 |= DELL ENTERPRISE NVME . 61500 |======== SAMSUNG ZFS MIRROR ... 353917 |================================================ Flexible IO Tester 3.25 Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory IOPS > Higher Is Better 10 Per VDEV RaidZ2 ... 146933 |================================================ 5 Per VDEV RaidZ2 .... 131333 |=========================================== Mirrors .............. 132600 |=========================================== DELL ENT NVME SSD1 ... 1553 |= DELL SAMSUNG SSD4 .... 17133 |====== DELL ENTERPRISE NVME . 9457 |=== SAMSUNG ZFS MIRROR ... 54433 |================== Flexible IO Tester 3.25 Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory MB/s > Higher Is Better 10 Per VDEV RaidZ2 ... 9131 |================================================== DELL ENT NVME SSD1 ... 203 |= DELL SAMSUNG SSD4 .... 2080 |=========== DELL ENTERPRISE NVME . 2285 |============= SAMSUNG ZFS MIRROR ... 6145 |================================== Flexible IO Tester 3.25 Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory IOPS > Higher Is Better 10 Per VDEV RaidZ2 ... 73033 |==================================== 5 Per VDEV RaidZ2 .... 91300 |============================================= Mirrors .............. 99900 |================================================= DELL ENT NVME SSD1 ... 1623 |= DELL SAMSUNG SSD4 .... 16633 |======== DELL ENTERPRISE NVME . 18300 |========= SAMSUNG ZFS MIRROR ... 49140 |======================== Flexible IO Tester 3.25 Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory MB/s > Higher Is Better DELL ENT NVME SSD1 ... 196 |= DELL SAMSUNG SSD4 .... 791 |===== DELL ENTERPRISE NVME . 955 |======= SAMSUNG ZFS MIRROR ... 7340 |================================================== Flexible IO Tester 3.25 Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory MB/s > Higher Is Better DELL ENT NVME SSD1 ... 195 |= DELL SAMSUNG SSD4 .... 2138 |================ DELL ENTERPRISE NVME . 1182 |========= SAMSUNG ZFS MIRROR ... 6804 |==================================================