45drivesxl60 2 x AMD EPYC 7513 32-Core testing with a Dell PowerEdge R7525 [0590KW] (2.6.6 BIOS) and Matrox G200eW3 on Debian GNU/Linux 11 via the Phoronix Test Suite.
HTML result view exported from: https://openbenchmarking.org/result/2209200-NE-2110221TJ08&gru&sro .
45drivesxl60 Processor Motherboard Memory Disk Chipset Graphics Monitor Network OS Kernel Compiler File-System Vulkan Screen Resolution 10 Per VDEV RaidZ2 5 Per VDEV RaidZ2 Mirrors DELL ENT NVME SSD1 DELL SAMSUNG SSD4 DELL ENTERPRISE NVME SAMSUNG ZFS MIRROR Intel Xeon Silver 4216 (32 Cores) 45Drives XL60 0123456789 256GB 742117GB FreeBSD 12.2-RELEASE-p10 (x86_64) Clang 10.0.1 zfs 2 x AMD EPYC 7513 32-Core (64 Cores / 128 Threads) Dell PowerEdge R7525 [0590KW] (2.6.6 BIOS) AMD Starship/Matisse 8 x 64 GB DDR4-3200MT/s 36ASF8G72PZ-3G2B2 240GB DELLBOSS VD + 2 x 1920GB Dell Ent NVMe CM + 2 x 3841GB SAMSUNG MZQL23T8 Matrox G200eW3 DELL E172FP 4 x Intel X710 for 10GBASE-T + 2 x Broadcom NetXtreme BCM5720 2-port PCIe Debian GNU/Linux 11 5.15.30-2-pve (x86_64) 1.0.2 GCC 10.2.1 20210110 ext4 1280x1024 zfs OpenBenchmarking.org Kernel Details - DELL ENT NVME SSD1, DELL SAMSUNG SSD4, DELL ENTERPRISE NVME, SAMSUNG ZFS MIRROR: Transparent Huge Pages: madvise Compiler Details - DELL ENT NVME SSD1, DELL SAMSUNG SSD4, DELL ENTERPRISE NVME, SAMSUNG ZFS MIRROR: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-mutex --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-10-Km9U7s/gcc-10-10.2.1/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-10-Km9U7s/gcc-10-10.2.1/debian/tmp-gcn/usr,hsa --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Disk Details - DELL ENT NVME SSD1: MQ-DEADLINE / errors=remount-ro,relatime,rw / Block Size: 4096 - DELL SAMSUNG SSD4: MQ-DEADLINE / errors=remount-ro,relatime,rw / Block Size: 4096 - DELL ENTERPRISE NVME: MQ-DEADLINE / errors=remount-ro,relatime,rw / Block Size: 4096 - SAMSUNG ZFS MIRROR: MQ-DEADLINE / errors=remount-ro,relatime,rw / Block Size: 131072 Processor Details - DELL ENT NVME SSD1, DELL SAMSUNG SSD4, DELL ENTERPRISE NVME, SAMSUNG ZFS MIRROR: CPU Microcode: 0xa00115d Security Details - DELL ENT NVME SSD1, DELL SAMSUNG SSD4, DELL ENTERPRISE NVME, SAMSUNG ZFS MIRROR: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl and seccomp + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling + srbds: Not affected + tsx_async_abort: Not affected
45drivesxl60 fio: Rand Read - POSIX AIO - No - Yes - 4KB - Default Test Directory fio: Rand Write - POSIX AIO - No - Yes - 4KB - Default Test Directory fio: Rand Read - POSIX AIO - No - Yes - 128KB - Default Test Directory fio: Rand Write - POSIX AIO - No - Yes - 128KB - Default Test Directory fio: Seq Read - POSIX AIO - No - Yes - 4KB - Default Test Directory fio: Seq Write - POSIX AIO - No - Yes - 4KB - Default Test Directory fio: Seq Read - POSIX AIO - No - Yes - 128KB - Default Test Directory fio: Seq Write - POSIX AIO - No - Yes - 128KB - Default Test Directory fio: Rand Read - POSIX AIO - No - Yes - 4KB - Default Test Directory fio: Rand Write - POSIX AIO - No - Yes - 4KB - Default Test Directory fio: Rand Write - POSIX AIO - No - Yes - 128KB - Default Test Directory fio: Seq Read - POSIX AIO - No - Yes - 4KB - Default Test Directory fio: Seq Write - POSIX AIO - No - Yes - 4KB - Default Test Directory fio: Seq Write - POSIX AIO - No - Yes - 128KB - Default Test Directory fio: Rand Read - POSIX AIO - No - Yes - 128KB - Default Test Directory fio: Seq Read - POSIX AIO - No - Yes - 128KB - Default Test Directory 10 Per VDEV RaidZ2 5 Per VDEV RaidZ2 Mirrors DELL ENT NVME SSD1 DELL SAMSUNG SSD4 DELL ENTERPRISE NVME SAMSUNG ZFS MIRROR 175333 92300 156667 72700 188000 127000 146933 73033 685 360 9083 735 495 9131 177333 110333 157000 93500 188000 135667 131333 91300 692 432 735 529 178333 129333 156333 95733 189333 137667 132600 99900 697 505 739 537 6991 18260 1561 1617 7183 17871 1553 1623 27.3 71.3 203 28.1 70.3 203 196 195 9825 6242 6327 16400 63067 6385 17133 16633 38.4 24.4 2054 246 24.3 2080 791 2138 12100 61233 7633 18300 22500 61500 9457 18300 47.2 240 2286 87.6 240 2285 955 1182 99133 180000 58733 49633 440133 353917 54433 49140 388 703 6210 1720 1382 6145 7340 6804 OpenBenchmarking.org
Flexible IO Tester Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.25 Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory 10 Per VDEV RaidZ2 5 Per VDEV RaidZ2 DELL ENT NVME SSD1 DELL ENTERPRISE NVME DELL SAMSUNG SSD4 Mirrors SAMSUNG ZFS MIRROR 40K 80K 120K 160K 200K SE +/- 666.67, N = 3 SE +/- 666.67, N = 3 SE +/- 14.33, N = 3 SE +/- 4.63, N = 3 SE +/- 666.67, N = 3 SE +/- 366.67, N = 3 175333 177333 6991 12100 9825 178333 99133
Flexible IO Tester Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.25 Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory 10 Per VDEV RaidZ2 5 Per VDEV RaidZ2 DELL ENT NVME SSD1 DELL ENTERPRISE NVME DELL SAMSUNG SSD4 Mirrors SAMSUNG ZFS MIRROR 40K 80K 120K 160K 200K SE +/- 416.33, N = 3 SE +/- 881.92, N = 3 SE +/- 222.92, N = 15 SE +/- 66.67, N = 3 SE +/- 1.86, N = 3 SE +/- 1452.97, N = 3 92300 110333 18260 61233 6242 129333 180000
Flexible IO Tester Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.25 Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory 10 Per VDEV RaidZ2 5 Per VDEV RaidZ2 DELL ENT NVME SSD1 DELL ENTERPRISE NVME DELL SAMSUNG SSD4 Mirrors SAMSUNG ZFS MIRROR 30K 60K 90K 120K 150K SE +/- 333.33, N = 3 SE +/- 577.35, N = 3 SE +/- 2.89, N = 3 SE +/- 3.79, N = 3 SE +/- 43.75, N = 3 SE +/- 333.33, N = 3 SE +/- 600.93, N = 3 156667 157000 1561 7633 6327 156333 58733
Flexible IO Tester Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.25 Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory 10 Per VDEV RaidZ2 5 Per VDEV RaidZ2 DELL ENT NVME SSD1 DELL ENTERPRISE NVME DELL SAMSUNG SSD4 Mirrors SAMSUNG ZFS MIRROR 20K 40K 60K 80K 100K SE +/- 862.17, N = 3 SE +/- 757.19, N = 3 SE +/- 10.41, N = 3 SE +/- 517.47, N = 3 SE +/- 120.19, N = 3 72700 93500 1617 18300 16400 95733 49633
Flexible IO Tester Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.25 Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory 10 Per VDEV RaidZ2 5 Per VDEV RaidZ2 DELL ENT NVME SSD1 DELL ENTERPRISE NVME DELL SAMSUNG SSD4 Mirrors SAMSUNG ZFS MIRROR 90K 180K 270K 360K 450K SE +/- 577.35, N = 3 SE +/- 577.35, N = 3 SE +/- 11.72, N = 3 SE +/- 33.33, N = 3 SE +/- 333.33, N = 3 SE +/- 10885.49, N = 15 188000 188000 7183 22500 63067 189333 440133
Flexible IO Tester Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.25 Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory 10 Per VDEV RaidZ2 5 Per VDEV RaidZ2 DELL ENT NVME SSD1 DELL ENTERPRISE NVME DELL SAMSUNG SSD4 Mirrors SAMSUNG ZFS MIRROR 80K 160K 240K 320K 400K SE +/- 333.33, N = 3 SE +/- 180.48, N = 14 SE +/- 57.74, N = 3 SE +/- 1.20, N = 3 SE +/- 333.33, N = 3 SE +/- 5411.07, N = 12 127000 135667 17871 61500 6385 137667 353917
Flexible IO Tester Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.25 Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory 10 Per VDEV RaidZ2 5 Per VDEV RaidZ2 DELL ENT NVME SSD1 DELL ENTERPRISE NVME DELL SAMSUNG SSD4 Mirrors SAMSUNG ZFS MIRROR 30K 60K 90K 120K 150K SE +/- 2369.38, N = 15 SE +/- 333.33, N = 3 SE +/- 4.10, N = 3 SE +/- 4.33, N = 3 SE +/- 33.33, N = 3 SE +/- 1400.00, N = 5 SE +/- 693.62, N = 3 146933 131333 1553 9457 17133 132600 54433
Flexible IO Tester Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.25 Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory 10 Per VDEV RaidZ2 5 Per VDEV RaidZ2 DELL ENT NVME SSD1 DELL ENTERPRISE NVME DELL SAMSUNG SSD4 Mirrors SAMSUNG ZFS MIRROR 20K 40K 60K 80K 100K SE +/- 924.36, N = 3 SE +/- 709.46, N = 3 SE +/- 4.84, N = 3 SE +/- 33.33, N = 3 SE +/- 1081.67, N = 3 SE +/- 376.59, N = 10 73033 91300 1623 18300 16633 99900 49140
Flexible IO Tester Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.25 Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory 10 Per VDEV RaidZ2 5 Per VDEV RaidZ2 DELL ENT NVME SSD1 DELL ENTERPRISE NVME DELL SAMSUNG SSD4 Mirrors SAMSUNG ZFS MIRROR 150 300 450 600 750 SE +/- 3.06, N = 3 SE +/- 2.67, N = 3 SE +/- 0.06, N = 3 SE +/- 0.00, N = 3 SE +/- 0.00, N = 3 SE +/- 2.85, N = 3 SE +/- 1.33, N = 3 685.0 692.0 27.3 47.2 38.4 697.0 388.0
Flexible IO Tester Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.25 Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory 10 Per VDEV RaidZ2 5 Per VDEV RaidZ2 DELL ENT NVME SSD1 DELL ENTERPRISE NVME DELL SAMSUNG SSD4 Mirrors SAMSUNG ZFS MIRROR 150 300 450 600 750 SE +/- 1.76, N = 3 SE +/- 2.33, N = 3 SE +/- 0.86, N = 15 SE +/- 0.33, N = 3 SE +/- 0.00, N = 3 SE +/- 5.61, N = 3 SE +/- 0.67, N = 3 360.0 432.0 71.3 240.0 24.4 505.0 703.0
Flexible IO Tester Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.25 Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory 10 Per VDEV RaidZ2 DELL ENT NVME SSD1 DELL ENTERPRISE NVME DELL SAMSUNG SSD4 SAMSUNG ZFS MIRROR 2K 4K 6K 8K 10K SE +/- 108.58, N = 3 SE +/- 1.45, N = 3 SE +/- 1.53, N = 3 SE +/- 15.21, N = 3 9083 203 2286 2054 6210
Flexible IO Tester Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.25 Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory 10 Per VDEV RaidZ2 5 Per VDEV RaidZ2 DELL ENT NVME SSD1 DELL ENTERPRISE NVME DELL SAMSUNG SSD4 Mirrors SAMSUNG ZFS MIRROR 400 800 1200 1600 2000 SE +/- 1.76, N = 3 SE +/- 1.76, N = 3 SE +/- 0.06, N = 3 SE +/- 0.32, N = 3 SE +/- 0.33, N = 3 SE +/- 1.67, N = 3 SE +/- 42.44, N = 15 735.0 735.0 28.1 87.6 246.0 739.0 1720.0
Flexible IO Tester Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.25 Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory 10 Per VDEV RaidZ2 5 Per VDEV RaidZ2 DELL ENT NVME SSD1 DELL ENTERPRISE NVME DELL SAMSUNG SSD4 Mirrors SAMSUNG ZFS MIRROR 300 600 900 1200 1500 SE +/- 0.73, N = 14 SE +/- 0.30, N = 3 SE +/- 1.20, N = 3 SE +/- 21.11, N = 12 495.0 529.0 70.3 240.0 24.3 537.0 1382.0
Flexible IO Tester Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.25 Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory 10 Per VDEV RaidZ2 DELL ENT NVME SSD1 DELL ENTERPRISE NVME DELL SAMSUNG SSD4 SAMSUNG ZFS MIRROR 2K 4K 6K 8K 10K SE +/- 114.13, N = 3 SE +/- 0.67, N = 3 SE +/- 0.67, N = 3 SE +/- 1.20, N = 3 SE +/- 46.95, N = 10 9131 203 2285 2080 6145
Flexible IO Tester Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.25 Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory DELL ENT NVME SSD1 DELL ENTERPRISE NVME DELL SAMSUNG SSD4 SAMSUNG ZFS MIRROR 1600 3200 4800 6400 8000 SE +/- 0.33, N = 3 SE +/- 0.33, N = 3 SE +/- 5.24, N = 3 SE +/- 74.71, N = 3 196 955 791 7340 1. (CC) gcc options: -rdynamic -ll -lnuma -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.25 Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory DELL ENT NVME SSD1 DELL ENTERPRISE NVME DELL SAMSUNG SSD4 SAMSUNG ZFS MIRROR 1500 3000 4500 6000 7500 SE +/- 0.67, N = 3 SE +/- 4.18, N = 3 SE +/- 88.50, N = 3 195 1182 2138 6804 1. (CC) gcc options: -rdynamic -ll -lnuma -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Phoronix Test Suite v10.8.4