45drivesxl60 2 x AMD EPYC 7513 32-Core testing with a Dell PowerEdge R7525 [0590KW] (2.6.6 BIOS) and Matrox G200eW3 on Debian GNU/Linux 11 via the Phoronix Test Suite.
HTML result view exported from: https://openbenchmarking.org/result/2209200-NE-2110221TJ08&sor&grr .
45drivesxl60 Processor Motherboard Memory Disk Chipset Graphics Monitor Network OS Kernel Compiler File-System Vulkan Screen Resolution 10 Per VDEV RaidZ2 5 Per VDEV RaidZ2 Mirrors DELL ENT NVME SSD1 DELL SAMSUNG SSD4 DELL ENTERPRISE NVME SAMSUNG ZFS MIRROR Intel Xeon Silver 4216 (32 Cores) 45Drives XL60 0123456789 256GB 742117GB FreeBSD 12.2-RELEASE-p10 (x86_64) Clang 10.0.1 zfs 2 x AMD EPYC 7513 32-Core (64 Cores / 128 Threads) Dell PowerEdge R7525 [0590KW] (2.6.6 BIOS) AMD Starship/Matisse 8 x 64 GB DDR4-3200MT/s 36ASF8G72PZ-3G2B2 240GB DELLBOSS VD + 2 x 1920GB Dell Ent NVMe CM + 2 x 3841GB SAMSUNG MZQL23T8 Matrox G200eW3 DELL E172FP 4 x Intel X710 for 10GBASE-T + 2 x Broadcom NetXtreme BCM5720 2-port PCIe Debian GNU/Linux 11 5.15.30-2-pve (x86_64) 1.0.2 GCC 10.2.1 20210110 ext4 1280x1024 zfs OpenBenchmarking.org Kernel Details - DELL ENT NVME SSD1, DELL SAMSUNG SSD4, DELL ENTERPRISE NVME, SAMSUNG ZFS MIRROR: Transparent Huge Pages: madvise Compiler Details - DELL ENT NVME SSD1, DELL SAMSUNG SSD4, DELL ENTERPRISE NVME, SAMSUNG ZFS MIRROR: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-mutex --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-10-Km9U7s/gcc-10-10.2.1/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-10-Km9U7s/gcc-10-10.2.1/debian/tmp-gcn/usr,hsa --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Disk Details - DELL ENT NVME SSD1: MQ-DEADLINE / errors=remount-ro,relatime,rw / Block Size: 4096 - DELL SAMSUNG SSD4: MQ-DEADLINE / errors=remount-ro,relatime,rw / Block Size: 4096 - DELL ENTERPRISE NVME: MQ-DEADLINE / errors=remount-ro,relatime,rw / Block Size: 4096 - SAMSUNG ZFS MIRROR: MQ-DEADLINE / errors=remount-ro,relatime,rw / Block Size: 131072 Processor Details - DELL ENT NVME SSD1, DELL SAMSUNG SSD4, DELL ENTERPRISE NVME, SAMSUNG ZFS MIRROR: CPU Microcode: 0xa00115d Security Details - DELL ENT NVME SSD1, DELL SAMSUNG SSD4, DELL ENTERPRISE NVME, SAMSUNG ZFS MIRROR: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl and seccomp + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling + srbds: Not affected + tsx_async_abort: Not affected
45drivesxl60 fio: Seq Write - POSIX AIO - No - Yes - 4KB - Default Test Directory fio: Seq Write - POSIX AIO - No - Yes - 4KB - Default Test Directory fio: Seq Read - POSIX AIO - No - Yes - 128KB - Default Test Directory fio: Seq Read - POSIX AIO - No - Yes - 4KB - Default Test Directory fio: Seq Read - POSIX AIO - No - Yes - 4KB - Default Test Directory fio: Rand Write - POSIX AIO - No - Yes - 4KB - Default Test Directory fio: Rand Write - POSIX AIO - No - Yes - 4KB - Default Test Directory fio: Seq Write - POSIX AIO - No - Yes - 128KB - Default Test Directory fio: Seq Write - POSIX AIO - No - Yes - 128KB - Default Test Directory fio: Rand Read - POSIX AIO - No - Yes - 4KB - Default Test Directory fio: Rand Read - POSIX AIO - No - Yes - 4KB - Default Test Directory fio: Rand Read - POSIX AIO - No - Yes - 128KB - Default Test Directory fio: Rand Read - POSIX AIO - No - Yes - 128KB - Default Test Directory fio: Seq Read - POSIX AIO - No - Yes - 128KB - Default Test Directory fio: Rand Write - POSIX AIO - No - Yes - 128KB - Default Test Directory fio: Rand Write - POSIX AIO - No - Yes - 128KB - Default Test Directory 10 Per VDEV RaidZ2 5 Per VDEV RaidZ2 Mirrors DELL ENT NVME SSD1 DELL SAMSUNG SSD4 DELL ENTERPRISE NVME SAMSUNG ZFS MIRROR 127000 495 146933 188000 735 92300 360 9131 73033 175333 685 156667 72700 9083 135667 529 131333 188000 735 110333 432 91300 177333 692 157000 93500 137667 537 132600 189333 739 129333 505 99900 178333 697 156333 95733 17871 70.3 1553 7183 28.1 18260 71.3 203 1623 6991 27.3 1561 196 195 1617 203 6385 24.3 17133 63067 246 6242 24.4 2080 16633 9825 38.4 6327 791 2138 16400 2054 61500 240 9457 22500 87.6 61233 240 2285 18300 12100 47.2 7633 955 1182 18300 2286 353917 1382 54433 440133 1720 180000 703 6145 49140 99133 388 58733 7340 6804 49633 6210 OpenBenchmarking.org
Flexible IO Tester Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.25 Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory SAMSUNG ZFS MIRROR Mirrors 5 Per VDEV RaidZ2 10 Per VDEV RaidZ2 DELL ENTERPRISE NVME DELL ENT NVME SSD1 DELL SAMSUNG SSD4 80K 160K 240K 320K 400K SE +/- 5411.07, N = 12 SE +/- 333.33, N = 3 SE +/- 333.33, N = 3 SE +/- 57.74, N = 3 SE +/- 180.48, N = 14 SE +/- 1.20, N = 3 353917 137667 135667 127000 61500 17871 6385
Flexible IO Tester Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.25 Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory SAMSUNG ZFS MIRROR Mirrors 5 Per VDEV RaidZ2 10 Per VDEV RaidZ2 DELL ENTERPRISE NVME DELL ENT NVME SSD1 DELL SAMSUNG SSD4 300 600 900 1200 1500 SE +/- 21.11, N = 12 SE +/- 1.20, N = 3 SE +/- 0.73, N = 14 SE +/- 0.30, N = 3 1382.0 537.0 529.0 495.0 240.0 70.3 24.3
Flexible IO Tester Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.25 Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory 10 Per VDEV RaidZ2 Mirrors 5 Per VDEV RaidZ2 SAMSUNG ZFS MIRROR DELL SAMSUNG SSD4 DELL ENTERPRISE NVME DELL ENT NVME SSD1 30K 60K 90K 120K 150K SE +/- 2369.38, N = 15 SE +/- 1400.00, N = 5 SE +/- 333.33, N = 3 SE +/- 693.62, N = 3 SE +/- 33.33, N = 3 SE +/- 4.33, N = 3 SE +/- 4.10, N = 3 146933 132600 131333 54433 17133 9457 1553
Flexible IO Tester Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.25 Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory SAMSUNG ZFS MIRROR Mirrors 5 Per VDEV RaidZ2 10 Per VDEV RaidZ2 DELL SAMSUNG SSD4 DELL ENTERPRISE NVME DELL ENT NVME SSD1 90K 180K 270K 360K 450K SE +/- 10885.49, N = 15 SE +/- 333.33, N = 3 SE +/- 577.35, N = 3 SE +/- 577.35, N = 3 SE +/- 33.33, N = 3 SE +/- 11.72, N = 3 440133 189333 188000 188000 63067 22500 7183
Flexible IO Tester Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.25 Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory SAMSUNG ZFS MIRROR Mirrors 5 Per VDEV RaidZ2 10 Per VDEV RaidZ2 DELL SAMSUNG SSD4 DELL ENTERPRISE NVME DELL ENT NVME SSD1 400 800 1200 1600 2000 SE +/- 42.44, N = 15 SE +/- 1.67, N = 3 SE +/- 1.76, N = 3 SE +/- 1.76, N = 3 SE +/- 0.33, N = 3 SE +/- 0.32, N = 3 SE +/- 0.06, N = 3 1720.0 739.0 735.0 735.0 246.0 87.6 28.1
Flexible IO Tester Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.25 Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory SAMSUNG ZFS MIRROR Mirrors 5 Per VDEV RaidZ2 10 Per VDEV RaidZ2 DELL ENTERPRISE NVME DELL ENT NVME SSD1 DELL SAMSUNG SSD4 40K 80K 120K 160K 200K SE +/- 1452.97, N = 3 SE +/- 881.92, N = 3 SE +/- 416.33, N = 3 SE +/- 66.67, N = 3 SE +/- 222.92, N = 15 SE +/- 1.86, N = 3 180000 129333 110333 92300 61233 18260 6242
Flexible IO Tester Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.25 Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory SAMSUNG ZFS MIRROR Mirrors 5 Per VDEV RaidZ2 10 Per VDEV RaidZ2 DELL ENTERPRISE NVME DELL ENT NVME SSD1 DELL SAMSUNG SSD4 150 300 450 600 750 SE +/- 0.67, N = 3 SE +/- 5.61, N = 3 SE +/- 2.33, N = 3 SE +/- 1.76, N = 3 SE +/- 0.33, N = 3 SE +/- 0.86, N = 15 SE +/- 0.00, N = 3 703.0 505.0 432.0 360.0 240.0 71.3 24.4
Flexible IO Tester Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.25 Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory 10 Per VDEV RaidZ2 SAMSUNG ZFS MIRROR DELL ENTERPRISE NVME DELL SAMSUNG SSD4 DELL ENT NVME SSD1 2K 4K 6K 8K 10K SE +/- 114.13, N = 3 SE +/- 46.95, N = 10 SE +/- 0.67, N = 3 SE +/- 1.20, N = 3 SE +/- 0.67, N = 3 9131 6145 2285 2080 203
Flexible IO Tester Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.25 Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory Mirrors 5 Per VDEV RaidZ2 10 Per VDEV RaidZ2 SAMSUNG ZFS MIRROR DELL ENTERPRISE NVME DELL SAMSUNG SSD4 DELL ENT NVME SSD1 20K 40K 60K 80K 100K SE +/- 1081.67, N = 3 SE +/- 709.46, N = 3 SE +/- 924.36, N = 3 SE +/- 376.59, N = 10 SE +/- 33.33, N = 3 SE +/- 4.84, N = 3 99900 91300 73033 49140 18300 16633 1623
Flexible IO Tester Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.25 Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory Mirrors 5 Per VDEV RaidZ2 10 Per VDEV RaidZ2 SAMSUNG ZFS MIRROR DELL ENTERPRISE NVME DELL SAMSUNG SSD4 DELL ENT NVME SSD1 40K 80K 120K 160K 200K SE +/- 666.67, N = 3 SE +/- 666.67, N = 3 SE +/- 666.67, N = 3 SE +/- 366.67, N = 3 SE +/- 4.63, N = 3 SE +/- 14.33, N = 3 178667 177333 175333 99133 12100 9825 6991
Flexible IO Tester Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.25 Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory Mirrors 5 Per VDEV RaidZ2 10 Per VDEV RaidZ2 SAMSUNG ZFS MIRROR DELL ENTERPRISE NVME DELL SAMSUNG SSD4 DELL ENT NVME SSD1 150 300 450 600 750 SE +/- 2.85, N = 3 SE +/- 2.67, N = 3 SE +/- 3.06, N = 3 SE +/- 1.33, N = 3 SE +/- 0.00, N = 3 SE +/- 0.00, N = 3 SE +/- 0.06, N = 3 697.0 692.0 685.0 388.0 47.2 38.4 27.3
Flexible IO Tester Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.25 Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory 5 Per VDEV RaidZ2 10 Per VDEV RaidZ2 Mirrors SAMSUNG ZFS MIRROR DELL ENTERPRISE NVME DELL SAMSUNG SSD4 DELL ENT NVME SSD1 30K 60K 90K 120K 150K SE +/- 577.35, N = 3 SE +/- 333.33, N = 3 SE +/- 333.33, N = 3 SE +/- 600.93, N = 3 SE +/- 3.79, N = 3 SE +/- 43.75, N = 3 SE +/- 2.89, N = 3 157000 156667 156333 58733 7633 6327 1561
Flexible IO Tester Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.25 Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory SAMSUNG ZFS MIRROR DELL ENTERPRISE NVME DELL SAMSUNG SSD4 DELL ENT NVME SSD1 1600 3200 4800 6400 8000 SE +/- 74.71, N = 3 SE +/- 0.33, N = 3 SE +/- 5.24, N = 3 SE +/- 0.33, N = 3 7340 955 791 196 1. (CC) gcc options: -rdynamic -ll -lnuma -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.25 Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory SAMSUNG ZFS MIRROR DELL SAMSUNG SSD4 DELL ENTERPRISE NVME DELL ENT NVME SSD1 1500 3000 4500 6000 7500 SE +/- 88.50, N = 3 SE +/- 4.18, N = 3 SE +/- 0.67, N = 3 6804 2138 1182 195 1. (CC) gcc options: -rdynamic -ll -lnuma -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.25 Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory Mirrors 5 Per VDEV RaidZ2 10 Per VDEV RaidZ2 SAMSUNG ZFS MIRROR DELL ENTERPRISE NVME DELL SAMSUNG SSD4 DELL ENT NVME SSD1 20K 40K 60K 80K 100K SE +/- 517.47, N = 3 SE +/- 757.19, N = 3 SE +/- 862.17, N = 3 SE +/- 120.19, N = 3 SE +/- 10.41, N = 3 95733 93500 72700 49633 18300 16400 1617
Flexible IO Tester Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.25 Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory 10 Per VDEV RaidZ2 SAMSUNG ZFS MIRROR DELL ENTERPRISE NVME DELL SAMSUNG SSD4 DELL ENT NVME SSD1 2K 4K 6K 8K 10K SE +/- 108.58, N = 3 SE +/- 15.21, N = 3 SE +/- 1.53, N = 3 SE +/- 1.45, N = 3 9083 6210 2286 2054 203
Phoronix Test Suite v10.8.4