45drivesxl60

2 x AMD EPYC 7513 32-Core testing with a Dell PowerEdge R7525 [0590KW] (2.6.6 BIOS) and Matrox G200eW3 on Debian GNU/Linux 11 via the Phoronix Test Suite.

HTML result view exported from: https://openbenchmarking.org/result/2209200-NE-2110221TJ08&grt&rdt.

45drivesxl60ProcessorMotherboardMemoryDiskChipsetGraphicsMonitorNetworkOSKernelCompilerFile-SystemVulkanScreen Resolution10 Per VDEV RaidZ25 Per VDEV RaidZ2MirrorsDELL ENT NVME SSD1DELL SAMSUNG SSD4DELL ENTERPRISE NVMESAMSUNG ZFS MIRRORIntel Xeon Silver 4216 (32 Cores)45Drives XL60 0123456789256GB742117GBFreeBSD12.2-RELEASE-p10 (x86_64)Clang 10.0.1zfs2 x AMD EPYC 7513 32-Core (64 Cores / 128 Threads)Dell PowerEdge R7525 [0590KW] (2.6.6 BIOS)AMD Starship/Matisse8 x 64 GB DDR4-3200MT/s 36ASF8G72PZ-3G2B2240GB DELLBOSS VD + 2 x 1920GB Dell Ent NVMe CM + 2 x 3841GB SAMSUNG MZQL23T8Matrox G200eW3DELL E172FP4 x Intel X710 for 10GBASE-T + 2 x Broadcom NetXtreme BCM5720 2-port PCIeDebian GNU/Linux 115.15.30-2-pve (x86_64)1.0.2GCC 10.2.1 20210110ext41280x1024zfsOpenBenchmarking.orgKernel Details- DELL ENT NVME SSD1, DELL SAMSUNG SSD4, DELL ENTERPRISE NVME, SAMSUNG ZFS MIRROR: Transparent Huge Pages: madviseCompiler Details- DELL ENT NVME SSD1, DELL SAMSUNG SSD4, DELL ENTERPRISE NVME, SAMSUNG ZFS MIRROR: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-mutex --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-10-Km9U7s/gcc-10-10.2.1/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-10-Km9U7s/gcc-10-10.2.1/debian/tmp-gcn/usr,hsa --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Disk Details- DELL ENT NVME SSD1: MQ-DEADLINE / errors=remount-ro,relatime,rw / Block Size: 4096- DELL SAMSUNG SSD4: MQ-DEADLINE / errors=remount-ro,relatime,rw / Block Size: 4096- DELL ENTERPRISE NVME: MQ-DEADLINE / errors=remount-ro,relatime,rw / Block Size: 4096- SAMSUNG ZFS MIRROR: MQ-DEADLINE / errors=remount-ro,relatime,rw / Block Size: 131072Processor Details- DELL ENT NVME SSD1, DELL SAMSUNG SSD4, DELL ENTERPRISE NVME, SAMSUNG ZFS MIRROR: CPU Microcode: 0xa00115dSecurity Details- DELL ENT NVME SSD1, DELL SAMSUNG SSD4, DELL ENTERPRISE NVME, SAMSUNG ZFS MIRROR: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl and seccomp + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling + srbds: Not affected + tsx_async_abort: Not affected

45drivesxl60fio: Rand Read - POSIX AIO - No - Yes - 4KB - Default Test Directoryfio: Rand Read - POSIX AIO - No - Yes - 4KB - Default Test Directoryfio: Rand Write - POSIX AIO - No - Yes - 4KB - Default Test Directoryfio: Rand Write - POSIX AIO - No - Yes - 4KB - Default Test Directoryfio: Rand Read - POSIX AIO - No - Yes - 128KB - Default Test Directoryfio: Rand Write - POSIX AIO - No - Yes - 128KB - Default Test Directoryfio: Rand Write - POSIX AIO - No - Yes - 128KB - Default Test Directoryfio: Seq Read - POSIX AIO - No - Yes - 4KB - Default Test Directoryfio: Seq Read - POSIX AIO - No - Yes - 4KB - Default Test Directoryfio: Seq Write - POSIX AIO - No - Yes - 4KB - Default Test Directoryfio: Seq Write - POSIX AIO - No - Yes - 4KB - Default Test Directoryfio: Seq Read - POSIX AIO - No - Yes - 128KB - Default Test Directoryfio: Seq Write - POSIX AIO - No - Yes - 128KB - Default Test Directoryfio: Seq Write - POSIX AIO - No - Yes - 128KB - Default Test Directoryfio: Rand Read - POSIX AIO - No - Yes - 128KB - Default Test Directoryfio: Seq Read - POSIX AIO - No - Yes - 128KB - Default Test Directory10 Per VDEV RaidZ25 Per VDEV RaidZ2MirrorsDELL ENT NVME SSD1DELL SAMSUNG SSD4DELL ENTERPRISE NVMESAMSUNG ZFS MIRROR685175333360923001566679083727007351880004951270001469339131730336921773334321103331570009350073518800052913566713133391300697178667505129333156333957337391893335371376671326009990027.3699171.3182601561203161728.1718370.3178711553203162319619538.4982524.4624263272054164002466306724.3638517133208016633791213847.21210024061233763322861830087.62250024061500945722851830095511823889913370318000058733621049633172044013313823539175443361454914073406804OpenBenchmarking.org

Flexible IO Tester

Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.25Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory10 Per VDEV RaidZ25 Per VDEV RaidZ2MirrorsDELL ENT NVME SSD1DELL SAMSUNG SSD4DELL ENTERPRISE NVMESAMSUNG ZFS MIRROR150300450600750SE +/- 3.06, N = 3SE +/- 2.67, N = 3SE +/- 2.85, N = 3SE +/- 0.06, N = 3SE +/- 0.00, N = 3SE +/- 0.00, N = 3SE +/- 1.33, N = 3685.0692.0697.027.338.447.2388.0

Flexible IO Tester

Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.25Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryMirrors10 Per VDEV RaidZ25 Per VDEV RaidZ2DELL ENT NVME SSD1DELL SAMSUNG SSD4DELL ENTERPRISE NVMESAMSUNG ZFS MIRROR40K80K120K160K200KSE +/- 666.67, N = 3SE +/- 666.67, N = 3SE +/- 666.67, N = 3SE +/- 14.33, N = 3SE +/- 4.63, N = 3SE +/- 366.67, N = 3178333175333177333699198251210099133

Flexible IO Tester

Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.25Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory10 Per VDEV RaidZ25 Per VDEV RaidZ2MirrorsDELL ENT NVME SSD1DELL SAMSUNG SSD4DELL ENTERPRISE NVMESAMSUNG ZFS MIRROR150300450600750SE +/- 1.76, N = 3SE +/- 2.33, N = 3SE +/- 5.61, N = 3SE +/- 0.86, N = 15SE +/- 0.00, N = 3SE +/- 0.33, N = 3SE +/- 0.67, N = 3360.0432.0505.071.324.4240.0703.0

Flexible IO Tester

Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.25Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory10 Per VDEV RaidZ25 Per VDEV RaidZ2MirrorsDELL ENT NVME SSD1DELL SAMSUNG SSD4DELL ENTERPRISE NVMESAMSUNG ZFS MIRROR40K80K120K160K200KSE +/- 416.33, N = 3SE +/- 881.92, N = 3SE +/- 1452.97, N = 3SE +/- 222.92, N = 15SE +/- 1.86, N = 3SE +/- 66.67, N = 39230011033312933318260624261233180000

Flexible IO Tester

Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.25Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory10 Per VDEV RaidZ25 Per VDEV RaidZ2MirrorsDELL ENT NVME SSD1DELL SAMSUNG SSD4DELL ENTERPRISE NVMESAMSUNG ZFS MIRROR30K60K90K120K150KSE +/- 333.33, N = 3SE +/- 577.35, N = 3SE +/- 333.33, N = 3SE +/- 2.89, N = 3SE +/- 43.75, N = 3SE +/- 3.79, N = 3SE +/- 600.93, N = 315666715700015633315616327763358733

Flexible IO Tester

Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.25Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory10 Per VDEV RaidZ2DELL ENT NVME SSD1DELL SAMSUNG SSD4DELL ENTERPRISE NVMESAMSUNG ZFS MIRROR2K4K6K8K10KSE +/- 108.58, N = 3SE +/- 1.45, N = 3SE +/- 1.53, N = 3SE +/- 15.21, N = 39083203205422866210

Flexible IO Tester

Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.25Type: Random Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory10 Per VDEV RaidZ25 Per VDEV RaidZ2MirrorsDELL ENT NVME SSD1DELL SAMSUNG SSD4DELL ENTERPRISE NVMESAMSUNG ZFS MIRROR20K40K60K80K100KSE +/- 862.17, N = 3SE +/- 757.19, N = 3SE +/- 517.47, N = 3SE +/- 10.41, N = 3SE +/- 120.19, N = 37270093500957331617164001830049633

Flexible IO Tester

Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.25Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory10 Per VDEV RaidZ25 Per VDEV RaidZ2MirrorsDELL ENT NVME SSD1DELL SAMSUNG SSD4DELL ENTERPRISE NVMESAMSUNG ZFS MIRROR400800120016002000SE +/- 1.76, N = 3SE +/- 1.76, N = 3SE +/- 1.67, N = 3SE +/- 0.06, N = 3SE +/- 0.33, N = 3SE +/- 0.32, N = 3SE +/- 42.44, N = 15735.0735.0739.028.1246.087.61720.0

Flexible IO Tester

Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.25Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory10 Per VDEV RaidZ25 Per VDEV RaidZ2MirrorsDELL ENT NVME SSD1DELL SAMSUNG SSD4DELL ENTERPRISE NVMESAMSUNG ZFS MIRROR90K180K270K360K450KSE +/- 577.35, N = 3SE +/- 577.35, N = 3SE +/- 333.33, N = 3SE +/- 11.72, N = 3SE +/- 33.33, N = 3SE +/- 10885.49, N = 1518800018800018933371836306722500440133

Flexible IO Tester

Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.25Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory10 Per VDEV RaidZ25 Per VDEV RaidZ2MirrorsDELL ENT NVME SSD1DELL SAMSUNG SSD4DELL ENTERPRISE NVMESAMSUNG ZFS MIRROR30060090012001500SE +/- 1.20, N = 3SE +/- 0.73, N = 14SE +/- 0.30, N = 3SE +/- 21.11, N = 12495.0529.0537.070.324.3240.01382.0

Flexible IO Tester

Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.25Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory10 Per VDEV RaidZ25 Per VDEV RaidZ2MirrorsDELL ENT NVME SSD1DELL SAMSUNG SSD4DELL ENTERPRISE NVMESAMSUNG ZFS MIRROR80K160K240K320K400KSE +/- 333.33, N = 3SE +/- 333.33, N = 3SE +/- 180.48, N = 14SE +/- 1.20, N = 3SE +/- 57.74, N = 3SE +/- 5411.07, N = 1212700013566713766717871638561500353917

Flexible IO Tester

Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.25Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory10 Per VDEV RaidZ25 Per VDEV RaidZ2MirrorsDELL ENT NVME SSD1DELL SAMSUNG SSD4DELL ENTERPRISE NVMESAMSUNG ZFS MIRROR30K60K90K120K150KSE +/- 2369.38, N = 15SE +/- 333.33, N = 3SE +/- 1400.00, N = 5SE +/- 4.10, N = 3SE +/- 33.33, N = 3SE +/- 4.33, N = 3SE +/- 693.62, N = 3146933131333132600155317133945754433

Flexible IO Tester

Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.25Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory10 Per VDEV RaidZ2DELL ENT NVME SSD1DELL SAMSUNG SSD4DELL ENTERPRISE NVMESAMSUNG ZFS MIRROR2K4K6K8K10KSE +/- 114.13, N = 3SE +/- 0.67, N = 3SE +/- 1.20, N = 3SE +/- 0.67, N = 3SE +/- 46.95, N = 109131203208022856145

Flexible IO Tester

Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.25Type: Sequential Write - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory10 Per VDEV RaidZ25 Per VDEV RaidZ2MirrorsDELL ENT NVME SSD1DELL SAMSUNG SSD4DELL ENTERPRISE NVMESAMSUNG ZFS MIRROR20K40K60K80K100KSE +/- 924.36, N = 3SE +/- 709.46, N = 3SE +/- 1081.67, N = 3SE +/- 4.84, N = 3SE +/- 33.33, N = 3SE +/- 376.59, N = 107303391300999001623166331830049140

Flexible IO Tester

Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.25Type: Random Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test DirectoryDELL ENT NVME SSD1DELL SAMSUNG SSD4DELL ENTERPRISE NVMESAMSUNG ZFS MIRROR16003200480064008000SE +/- 0.33, N = 3SE +/- 5.24, N = 3SE +/- 0.33, N = 3SE +/- 74.71, N = 319679195573401. (CC) gcc options: -rdynamic -ll -lnuma -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

Flexible IO Tester

Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.25Type: Sequential Read - Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 128KB - Disk Target: Default Test DirectoryDELL ENT NVME SSD1DELL SAMSUNG SSD4DELL ENTERPRISE NVMESAMSUNG ZFS MIRROR15003000450060007500SE +/- 4.18, N = 3SE +/- 0.67, N = 3SE +/- 88.50, N = 31952138118268041. (CC) gcc options: -rdynamic -ll -lnuma -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native


Phoronix Test Suite v10.8.4