9disk-raidz, JHMultidisk-ZFS-RaidZ-JBod, zfs-raidz2-new

ZFS RaidZ, 9-drive JBoD, qla-speed=2Gbx2 (multi-pathed), Atime=off JHMultidisk-ZFS-RaidZ-JBod: ZFS RaidZ, 5-drive JBoD, qla-speed=2Gbx2 (multi-pathed), Atime=off zfs-raidz2-new: ZFS-on 10-disk RaidZ2, qla-speed=2Gbx2 (multi-pathed), NoAtime

HTML result view exported from: https://openbenchmarking.org/result/1304291-FO-MERGE284129,1304257-FO-MERGE786390,1304271-FO-MERGE184123&obr_sor=y.

9disk-raidz, JHMultidisk-ZFS-RaidZ-JBod, zfs-raidz2-newProcessorMotherboardChipsetMemoryDiskGraphicsNetworkOSKernelCompilerFile-SystemJHMultidisk-ZFS-9disk-RaidZ-JBoDJHMultidisk-EXT4-on-9disk-RaidZ-zvolJHMultidisk-Ext4-LVM-10disk-md-Raid6JHMultidisk-Ext4-LVM-10disk-md-Raid6-sJHMultidisk-EXT4-on-10disk-RaidZ2-zvolJHMultidisk-ZFS-RaidZ-JBoDJHMultidisk-ZFS-Raid10JHMultidisk-ZFS-Mirror-JBodJHMultidisk-EXT4-LVM-mdRaid5-JBoDJHMultidisk-EXT4-LVM-mdMirrorJHMultidisk-EXT4-LVM-HWRaid5JHMultidisk-ZFS-10disk-RaidZ2-JBoDzfs-raidz2-new: JHMultidisk-EXT4-on-10disk-RaidZ2-zvolIntel Xeon E3-1230 V2 @ 3.30GHz (8 Cores)IBM 00D3729Intel Xeon E3-1200 v2/Ivy4 x 4096 MB DDR3-1600MHz Hynix10 x 137GB DISK + 10 x 144GB DISK + 2 x 574GB RAID 5 + 500GB Seagate ST500NM0011 + 500GB Western Digital WD5003ABYX-23Matrox s G200eR2Intel 82574L Gigabit ConnectionRedHatEnterpriseServer 6.42.6.32-358.6.1.el6.x86_64 (x86_64)GCC 4.4.7 20120313ext410 x 137GB DISK + 10 x 144GB DISK + 500GB Seagate ST500NM0011 + 500GB Western Digital WD5003ABYX-232.6.32-358.2.1.el6.x86_64 (x86_64)500GB Seagate ST500NM0011 + 500GB Western Digital WD5003ABYX-23 + 10 x 137GB DISK + 10 x 144GB DISK + 2 x 574GB RAID 510 x 137GB DISK + 10 x 144GB DISK + 500GB Seagate ST500NM0011 + 500GB Western Digital WD5003ABYX-23500GB Seagate ST500NM0011 + 500GB Western Digital WD5003ABYX-23 + 10 x 137GB DISK + 10 x 144GB DISK + 2 x 574GB RAID 510 x 137GB DISK + 10 x 144GB DISK + 500GB Seagate ST500NM0011 + 500GB Western Digital WD5003ABYX-2310 x 137GB DISK + 10 x 144GB DISK + 2 x 574GB RAID 5 + 500GB Seagate ST500NM0011 + 500GB Western Digital WD5003ABYX-232.6.32-358.6.1.el6.x86_64 (x86_64)OpenBenchmarking.orgCompiler Details- --build=x86_64-redhat-linux --disable-dssi --disable-libjava-multilib --disable-libunwind-exceptions --enable-__cxa_atexit --enable-bootstrap --enable-checking=release --enable-gnu-unique-object --enable-java-awt=gtk --enable-java-maintainer-mode --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-libgcj-multifile --enable-shared --enable-threads=posix --mandir=/usr/share/man --with-arch_32=i686 --with-cloog --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre --with-ppl --with-tune=genericDisk Mount Options Details- barrier=1,data=ordered,relatime,rw

9disk-raidz, JHMultidisk-ZFS-RaidZ-JBod, zfs-raidz2-newfs-mark: 1000 Files, 1MB Sizefs-mark: 5000 Files, 1MB Size, 4 Threadsfs-mark: 4000 Files, 32 Sub Dirs, 1MB Sizefs-mark: 1000 Files, 1MB Size, No Sync/FSyncJHMultidisk-ZFS-9disk-RaidZ-JBoDJHMultidisk-EXT4-on-9disk-RaidZ-zvolJHMultidisk-Ext4-LVM-10disk-md-Raid6JHMultidisk-Ext4-LVM-10disk-md-Raid6-sJHMultidisk-EXT4-on-10disk-RaidZ2-zvolJHMultidisk-ZFS-RaidZ-JBoDJHMultidisk-ZFS-Raid10JHMultidisk-ZFS-Mirror-JBodJHMultidisk-EXT4-LVM-mdRaid5-JBoDJHMultidisk-EXT4-LVM-mdMirrorJHMultidisk-EXT4-LVM-HWRaid5JHMultidisk-ZFS-10disk-RaidZ2-JBoDzfs-raidz2-new: JHMultidisk-EXT4-on-10disk-RaidZ2-zvol34.8347.7034.23653.5219.4734.7020.78391.886.5318.876.701813.456.6319.906.771946.6816.4328.6317.33250.3725.6332.0024.67147.3324.4032.2723.77126.7012.4013.3012.4337.7011.8722.0711.772029.4013.2023.5713.101914.8314.5021.0714.871858.2824.9040.4027.77501.58OpenBenchmarking.org

FS-Mark

Test: 1000 Files, 1MB Size

OpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 1000 Files, 1MB SizeJHMultidisk-ZFS-9disk-RaidZ-JBoDJHMultidisk-ZFS-RaidZ-JBoDJHMultidisk-ZFS-10disk-RaidZ2-JBoDJHMultidisk-ZFS-Raid10JHMultidisk-EXT4-on-9disk-RaidZ-zvolJHMultidisk-EXT4-on-10disk-RaidZ2-zvolJHMultidisk-EXT4-LVM-HWRaid5JHMultidisk-EXT4-LVM-mdMirrorJHMultidisk-ZFS-Mirror-JBodJHMultidisk-EXT4-LVM-mdRaid5-JBoDJHMultidisk-Ext4-LVM-10disk-md-Raid6-sJHMultidisk-Ext4-LVM-10disk-md-Raid6816243240SE +/- 0.33, N = 3SE +/- 0.22, N = 3SE +/- 0.40, N = 4SE +/- 0.10, N = 3SE +/- 0.15, N = 3SE +/- 0.13, N = 3SE +/- 0.06, N = 3SE +/- 0.00, N = 3SE +/- 0.10, N = 3SE +/- 0.03, N = 3SE +/- 0.03, N = 3SE +/- 0.03, N = 334.8325.6324.9024.4019.4716.4314.5013.2012.4011.876.636.531. (CC) gcc options: -static

FS-Mark

Test: 5000 Files, 1MB Size, 4 Threads

OpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 5000 Files, 1MB Size, 4 ThreadsJHMultidisk-ZFS-9disk-RaidZ-JBoDJHMultidisk-ZFS-10disk-RaidZ2-JBoDJHMultidisk-EXT4-on-9disk-RaidZ-zvolJHMultidisk-ZFS-Raid10JHMultidisk-ZFS-RaidZ-JBoDJHMultidisk-EXT4-on-10disk-RaidZ2-zvolJHMultidisk-EXT4-LVM-mdMirrorJHMultidisk-EXT4-LVM-mdRaid5-JBoDJHMultidisk-EXT4-LVM-HWRaid5JHMultidisk-Ext4-LVM-10disk-md-Raid6-sJHMultidisk-Ext4-LVM-10disk-md-Raid6JHMultidisk-ZFS-Mirror-JBod1122334455SE +/- 0.15, N = 3SE +/- 0.40, N = 3SE +/- 0.38, N = 3SE +/- 0.13, N = 3SE +/- 0.23, N = 3SE +/- 0.35, N = 3SE +/- 0.42, N = 3SE +/- 0.07, N = 3SE +/- 0.13, N = 3SE +/- 0.00, N = 3SE +/- 0.38, N = 3SE +/- 0.10, N = 347.7040.4034.7032.2732.0028.6323.5722.0721.0719.9018.8713.301. (CC) gcc options: -static

FS-Mark

Test: 4000 Files, 32 Sub Dirs, 1MB Size

OpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 4000 Files, 32 Sub Dirs, 1MB SizeJHMultidisk-ZFS-9disk-RaidZ-JBoDJHMultidisk-ZFS-10disk-RaidZ2-JBoDJHMultidisk-ZFS-RaidZ-JBoDJHMultidisk-ZFS-Raid10JHMultidisk-EXT4-on-9disk-RaidZ-zvolJHMultidisk-EXT4-on-10disk-RaidZ2-zvolJHMultidisk-EXT4-LVM-HWRaid5JHMultidisk-EXT4-LVM-mdMirrorJHMultidisk-ZFS-Mirror-JBodJHMultidisk-EXT4-LVM-mdRaid5-JBoDJHMultidisk-Ext4-LVM-10disk-md-Raid6-sJHMultidisk-Ext4-LVM-10disk-md-Raid6816243240SE +/- 0.13, N = 3SE +/- 0.15, N = 3SE +/- 0.15, N = 3SE +/- 0.15, N = 3SE +/- 0.28, N = 6SE +/- 0.25, N = 4SE +/- 0.03, N = 3SE +/- 0.17, N = 3SE +/- 0.09, N = 3SE +/- 0.03, N = 3SE +/- 0.03, N = 3SE +/- 0.00, N = 334.2327.7724.6723.7720.7817.3314.8713.1012.4311.776.776.701. (CC) gcc options: -static

FS-Mark

Test: 1000 Files, 1MB Size, No Sync/FSync

OpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 1000 Files, 1MB Size, No Sync/FSyncJHMultidisk-EXT4-LVM-mdRaid5-JBoDJHMultidisk-Ext4-LVM-10disk-md-Raid6-sJHMultidisk-EXT4-LVM-mdMirrorJHMultidisk-EXT4-LVM-HWRaid5JHMultidisk-Ext4-LVM-10disk-md-Raid6JHMultidisk-ZFS-9disk-RaidZ-JBoDJHMultidisk-ZFS-10disk-RaidZ2-JBoDJHMultidisk-EXT4-on-9disk-RaidZ-zvolJHMultidisk-EXT4-on-10disk-RaidZ2-zvolJHMultidisk-ZFS-RaidZ-JBoDJHMultidisk-ZFS-Raid10JHMultidisk-ZFS-Mirror-JBod400800120016002000SE +/- 26.86, N = 3SE +/- 32.13, N = 4SE +/- 55.09, N = 6SE +/- 69.10, N = 6SE +/- 100.19, N = 6SE +/- 175.26, N = 6SE +/- 184.86, N = 6SE +/- 96.23, N = 6SE +/- 57.73, N = 6SE +/- 13.79, N = 6SE +/- 5.68, N = 6SE +/- 1.00, N = 62029.401946.681914.831858.281813.45653.52501.58391.88250.37147.33126.7037.701. (CC) gcc options: -static

Phoronix Test Suite v10.8.4