ZFS SSD Benchmark: RAW IMAGE vs QCOW2 vs ZVOL for KVM

FNAS

HTML result view exported from: https://openbenchmarking.org/result/1803243-KH-1803231TY25&rdt&grs.

ZFS SSD Benchmark: RAW IMAGE vs QCOW2 vs ZVOL for KVMProcessorMotherboardChipsetMemoryDiskGraphicsNetworkOSKernelCompilerFile-SystemScreen ResolutionSystem Layerzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolpve-ubuntu-fnas-18 x Intel Celeron_4x0 (Conroe/Merom Class Core 2) @ 1.86GHz (8 Cores)QEMU Standard PC (i440FX + PIIX 1996)Intel 440FX- 82441FX PMC1 x 1024 MB RAM QEMU54GB QEMU HDDCirrus Logic GD 5446Realtek RTL-8100/8101L/8139Debian 9.14.9.0-4-amd64 (x86_64)GCC 6.3.0 20170516ext41024x768qemuCommon KVM @ 3.09GHz (2 Cores)2048MB537GB QEMU HDDDevice 1234:1111Red Hat Virtio deviceUbuntu 16.044.4.0-116-generic (x86_64)GCC 5.4.0 20160609800x600OpenBenchmarking.orgCompiler Details- zfs ssd mirror, ashift=13, recordsize 8k, raw image: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 64k, raw image: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 8k, zvol: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 64k, zvol: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - pve-ubuntu-fnas-1: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --disable-werror --enable-checking=release --enable-clocale=gnu --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-tune=generic -v Disk Details- zfs ssd mirror, ashift=13, recordsize 8k, raw image: CFQ / data=ordered,errors=remount-ro,relatime,rw- zfs ssd mirror, ashift=13, recordsize 64k, raw image: CFQ / data=ordered,errors=remount-ro,relatime,rw- zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image: CFQ / data=ordered,errors=remount-ro,relatime,rw- zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image: CFQ / data=ordered,errors=remount-ro,relatime,rw- zfs ssd mirror, ashift=13, recordsize 8k, zvol: CFQ / data=ordered,errors=remount-ro,relatime,rw- zfs ssd mirror, ashift=13, recordsize 64k, zvol: CFQ / data=ordered,errors=remount-ro,relatime,rw- pve-ubuntu-fnas-1: DEADLINE / data=ordered,errors=remount-ro,relatime,rwSystem Details- zfs ssd mirror, ashift=13, recordsize 8k, raw image: Python 2.7.13.- zfs ssd mirror, ashift=13, recordsize 64k, raw image: Python 2.7.13.- zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image: Python 2.7.13.- zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image: Python 2.7.13.- zfs ssd mirror, ashift=13, recordsize 8k, zvol: Python 2.7.13.- zfs ssd mirror, ashift=13, recordsize 64k, zvol: Python 2.7.13.- pve-ubuntu-fnas-1: Disk Scheduler: DEADLINE. Python 2.7.12.

ZFS SSD Benchmark: RAW IMAGE vs QCOW2 vs ZVOL for KVMdbench: 256dbench: 1dbench: 6dbench: 12iozone: 64Kb - 8GB - Write Performanceiozone: 4Kb - 8GB - Write Performanceiozone: 1MB - 8GB - Read Performanceiozone: 64Kb - 8GB - Read Performanceiozone: 4Kb - 8GB - Read Performanceiozone: 1MB - 8GB - Write Performanceaio-stress: Rand Writecompilebench: Read Compiled Treecompilebench: Initial Createcompilebench: Compiledbench: 128dbench: 48zfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolpve-ubuntu-fnas-155.7742.1891.96137.6655.7257.00167.07172.38174.6360.0098.0155.7248.4072.1397.79181.38120.1842.60105.25166.12160.28162.04216.63226.37224.74157.79131.8164.8386.85147.18207.32292.2255.5742.2089.99137.5058.8761.43165.90174.56173.6865.2293.7056.7242.5461.6589.75176.41118.7042.33102.98162.59156.04161.59215.34225.08223.31156.30125.7464.6982.40142.88203.76294.0358.4740.8786.79128.7667.6274.0680.5084.8684.9572.07100.8746.5849.3291.6988.41153.6362.0042.7288.54133.5560.0363.1086.0788.9094.9966.79111.3360.7250.1181.59102.07177.89333.19179.20320.86387.51111.94111.83111.88112.04112.02111.941449.7764.91100.89117.05259.95437.60OpenBenchmarking.org

Dbench

Client Count: 256

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 256zfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolpve-ubuntu-fnas-170140210280350SE +/- 2.11, N = 6SE +/- 0.40, N = 3SE +/- 0.85, N = 3SE +/- 0.23, N = 3SE +/- 1.39, N = 6SE +/- 2.00, N = 6SE +/- 6.69, N = 355.77120.1855.57118.7058.4762.00333.191. (CC) gcc options: -lpopt -O2

Dbench

Client Count: 1

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 1zfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolpve-ubuntu-fnas-14080120160200SE +/- 0.05, N = 3SE +/- 0.09, N = 3SE +/- 0.02, N = 3SE +/- 0.08, N = 3SE +/- 0.15, N = 3SE +/- 0.01, N = 3SE +/- 0.06, N = 342.1842.6042.2042.3340.8742.72179.201. (CC) gcc options: -lpopt -O2

Flexible IO Tester

Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvol306090120150SE +/- 0.41, N = 3SE +/- 0.03, N = 3SE +/- 0.67, N = 3SE +/- 3.31, N = 6SE +/- 0.59, N = 374.27138.0070.77139.0036.4371.371. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Dbench

Client Count: 6

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 6zfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolpve-ubuntu-fnas-170140210280350SE +/- 1.36, N = 3SE +/- 1.07, N = 3SE +/- 0.90, N = 3SE +/- 0.91, N = 3SE +/- 0.16, N = 3SE +/- 1.27, N = 3SE +/- 6.43, N = 691.96105.2589.99102.9886.7988.54320.861. (CC) gcc options: -lpopt -O2

Flexible IO Tester

Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvol306090120150SE +/- 0.31, N = 3SE +/- 0.58, N = 3SE +/- 0.09, N = 3SE +/- 0.67, N = 3SE +/- 1.08, N = 6SE +/- 2.09, N = 671.20146.0070.23143.0042.0554.751. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvol306090120150SE +/- 0.93, N = 3SE +/- 1.08, N = 3SE +/- 2.33, N = 3SE +/- 6.42, N = 6SE +/- 1.94, N = 674.37152.0072.03147.0045.3366.281. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvol306090120150SE +/- 0.03, N = 3SE +/- 0.88, N = 3SE +/- 0.25, N = 3SE +/- 0.67, N = 3SE +/- 3.30, N = 6SE +/- 0.81, N = 361.33118.0059.40113.0038.7566.401. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Dbench

Client Count: 12

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 12zfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolpve-ubuntu-fnas-180160240320400SE +/- 1.52, N = 3SE +/- 0.58, N = 3SE +/- 1.47, N = 3SE +/- 0.75, N = 3SE +/- 0.84, N = 3SE +/- 0.41, N = 3SE +/- 2.88, N = 3137.66166.12137.50162.59128.76133.55387.511. (CC) gcc options: -lpopt -O2

IOzone

Record Size: 64Kb - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 8GB - Disk Test: Write Performancezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolpve-ubuntu-fnas-14080120160200SE +/- 0.37, N = 3SE +/- 2.94, N = 6SE +/- 0.92, N = 6SE +/- 2.73, N = 6SE +/- 3.14, N = 6SE +/- 1.77, N = 6SE +/- 0.04, N = 355.72160.2858.87156.0467.6260.03111.941. (CC) gcc options: -O3

IOzone

Record Size: 4Kb - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 8GB - Disk Test: Write Performancezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolpve-ubuntu-fnas-14080120160200SE +/- 1.60, N = 6SE +/- 1.03, N = 3SE +/- 1.02, N = 6SE +/- 1.79, N = 3SE +/- 1.13, N = 3SE +/- 1.08, N = 3SE +/- 0.03, N = 357.00162.0461.43161.5974.0663.10111.831. (CC) gcc options: -O3

IOzone

Record Size: 1MB - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 8GB - Disk Test: Read Performancezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolpve-ubuntu-fnas-150100150200250SE +/- 1.25, N = 3SE +/- 0.57, N = 3SE +/- 1.22, N = 3SE +/- 0.79, N = 3SE +/- 1.26, N = 3SE +/- 0.58, N = 3SE +/- 0.24, N = 3167.07216.63165.90215.3480.5086.07111.881. (CC) gcc options: -O3

IOzone

Record Size: 64Kb - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 8GB - Disk Test: Read Performancezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolpve-ubuntu-fnas-150100150200250SE +/- 0.49, N = 3SE +/- 0.36, N = 3SE +/- 0.49, N = 3SE +/- 0.72, N = 3SE +/- 2.08, N = 6SE +/- 1.37, N = 4SE +/- 0.08, N = 3172.38226.37174.56225.0884.8688.90112.041. (CC) gcc options: -O3

IOzone

Record Size: 4Kb - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 8GB - Disk Test: Read Performancezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolpve-ubuntu-fnas-150100150200250SE +/- 0.79, N = 3SE +/- 2.04, N = 3SE +/- 0.12, N = 3SE +/- 0.88, N = 3SE +/- 2.19, N = 6SE +/- 1.27, N = 3SE +/- 0.09, N = 3174.63224.74173.68223.3184.9594.99112.021. (CC) gcc options: -O3

IOzone

Record Size: 1MB - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 8GB - Disk Test: Write Performancezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolpve-ubuntu-fnas-1306090120150SE +/- 1.42, N = 6SE +/- 0.85, N = 3SE +/- 2.01, N = 6SE +/- 0.82, N = 3SE +/- 2.02, N = 6SE +/- 0.50, N = 3SE +/- 0.04, N = 360.00157.7965.22156.3072.0766.79111.941. (CC) gcc options: -O3

Flexible IO Tester

Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvol714212835SE +/- 0.20, N = 3SE +/- 0.26, N = 3SE +/- 0.19, N = 3SE +/- 0.24, N = 3SE +/- 0.10, N = 3SE +/- 0.21, N = 429.2017.2025.7315.9720.7012.051. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvol100200300400500SE +/- 0.67, N = 3SE +/- 0.67, N = 3SE +/- 1.20, N = 3SE +/- 10.48, N = 6SE +/- 5.61, N = 32292892262914724581. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvol100200300400500SE +/- 1.00, N = 3SE +/- 2.08, N = 3SE +/- 1.00, N = 3SE +/- 1.20, N = 3SE +/- 7.46, N = 6SE +/- 10.34, N = 62302942282924734541. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

AIO-Stress

Test: Random Write

OpenBenchmarking.orgMB/s, More Is BetterAIO-Stress 0.21Test: Random Writezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolpve-ubuntu-fnas-130060090012001500SE +/- 0.50, N = 3SE +/- 0.94, N = 3SE +/- 1.19, N = 3SE +/- 0.78, N = 3SE +/- 3.01, N = 6SE +/- 0.51, N = 3SE +/- 21.95, N = 598.01131.8193.70125.74100.87111.331449.771. (CC) gcc options: -pthread -laio

Flexible IO Tester

Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvol80160240320400SE +/- 1.33, N = 3SE +/- 1.76, N = 3SE +/- 0.58, N = 3SE +/- 1.33, N = 3SE +/- 4.51, N = 3SE +/- 7.51, N = 62523032443063533471. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Compile Bench

Test: Read Compiled Tree

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Read Compiled Treezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolpve-ubuntu-fnas-11428425670SE +/- 0.15, N = 3SE +/- 0.33, N = 3SE +/- 0.52, N = 3SE +/- 0.14, N = 3SE +/- 0.65, N = 3SE +/- 2.51, N = 3SE +/- 0.23, N = 355.7264.8356.7264.6946.5860.7264.91

Flexible IO Tester

Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvol918273645SE +/- 0.40, N = 3SE +/- 0.03, N = 3SE +/- 0.65, N = 4SE +/- 0.10, N = 3SE +/- 0.38, N = 3SE +/- 0.52, N = 336.8038.9340.6538.6039.2040.731. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvol50100150200250SE +/- 1.45, N = 3SE +/- 2.52, N = 3SE +/- 0.88, N = 3SE +/- 0.58, N = 32292392322372302301. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Compile Bench

Test: Initial Create

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Initial Createzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolpve-ubuntu-fnas-120406080100SE +/- 1.91, N = 3SE +/- 3.08, N = 3SE +/- 8.32, N = 3SE +/- 6.42, N = 3SE +/- 3.57, N = 3SE +/- 5.96, N = 3SE +/- 2.22, N = 348.4086.8542.5482.4049.3250.11100.89

Compile Bench

Test: Compile

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Compilezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolpve-ubuntu-fnas-1306090120150SE +/- 2.12, N = 6SE +/- 5.21, N = 6SE +/- 4.97, N = 6SE +/- 5.01, N = 6SE +/- 2.73, N = 6SE +/- 1.40, N = 3SE +/- 0.37, N = 372.13147.1861.65142.8891.6981.59117.05

Dbench

Client Count: 128

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 128zfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolpve-ubuntu-fnas-160120180240300SE +/- 5.85, N = 6SE +/- 2.00, N = 3SE +/- 5.76, N = 6SE +/- 0.99, N = 3SE +/- 4.52, N = 6SE +/- 4.96, N = 6SE +/- 6.29, N = 697.79207.3289.75203.7688.41102.07259.951. (CC) gcc options: -lpopt -O2

Dbench

Client Count: 48

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 48zfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolpve-ubuntu-fnas-190180270360450SE +/- 7.08, N = 6SE +/- 4.21, N = 3SE +/- 7.08, N = 6SE +/- 4.38, N = 3SE +/- 6.84, N = 6SE +/- 6.43, N = 6SE +/- 4.55, N = 3181.38292.22176.41294.03153.63177.89437.601. (CC) gcc options: -lpopt -O2

Flexible IO Tester

Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvol1122334455SE +/- 1.78, N = 6SE +/- 1.80, N = 6SE +/- 1.80, N = 6SE +/- 1.72, N = 6SE +/- 0.57, N = 6SE +/- 0.64, N = 641.8347.1032.7046.3024.1529.771. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvol1326395265SE +/- 2.11, N = 6SE +/- 0.23, N = 3SE +/- 1.52, N = 6SE +/- 1.33, N = 6SE +/- 1.10, N = 6SE +/- 0.32, N = 358.3255.5045.8554.1745.0244.731. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Phoronix Test Suite v10.8.4