ZFS SSD Benchmark: RAW IMAGE vs QCOW2 vs ZVOL for KVM

FNAS

HTML result view exported from: https://openbenchmarking.org/result/1803243-KH-1803231TY25&grt&sor.

ZFS SSD Benchmark: RAW IMAGE vs QCOW2 vs ZVOL for KVMProcessorMotherboardChipsetMemoryDiskGraphicsNetworkOSKernelCompilerFile-SystemScreen ResolutionSystem Layerzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvolpve-ubuntu-fnas-18 x Intel Celeron_4x0 (Conroe/Merom Class Core 2) @ 1.86GHz (8 Cores)QEMU Standard PC (i440FX + PIIX 1996)Intel 440FX- 82441FX PMC1 x 1024 MB RAM QEMU54GB QEMU HDDCirrus Logic GD 5446Realtek RTL-8100/8101L/8139Debian 9.14.9.0-4-amd64 (x86_64)GCC 6.3.0 20170516ext41024x768qemuCommon KVM @ 3.09GHz (2 Cores)2048MB537GB QEMU HDDDevice 1234:1111Red Hat Virtio deviceUbuntu 16.044.4.0-116-generic (x86_64)GCC 5.4.0 20160609800x600OpenBenchmarking.orgCompiler Details- zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 64k, raw image: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 64k, zvol: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 8k, raw image: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 8k, zvol: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - pve-ubuntu-fnas-1: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --disable-werror --enable-checking=release --enable-clocale=gnu --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-tune=generic -v Disk Details- zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image: CFQ / data=ordered,errors=remount-ro,relatime,rw- zfs ssd mirror, ashift=13, recordsize 64k, raw image: CFQ / data=ordered,errors=remount-ro,relatime,rw- zfs ssd mirror, ashift=13, recordsize 64k, zvol: CFQ / data=ordered,errors=remount-ro,relatime,rw- zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image: CFQ / data=ordered,errors=remount-ro,relatime,rw- zfs ssd mirror, ashift=13, recordsize 8k, raw image: CFQ / data=ordered,errors=remount-ro,relatime,rw- zfs ssd mirror, ashift=13, recordsize 8k, zvol: CFQ / data=ordered,errors=remount-ro,relatime,rw- pve-ubuntu-fnas-1: DEADLINE / data=ordered,errors=remount-ro,relatime,rwSystem Details- zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image: Python 2.7.13.- zfs ssd mirror, ashift=13, recordsize 64k, raw image: Python 2.7.13.- zfs ssd mirror, ashift=13, recordsize 64k, zvol: Python 2.7.13.- zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image: Python 2.7.13.- zfs ssd mirror, ashift=13, recordsize 8k, raw image: Python 2.7.13.- zfs ssd mirror, ashift=13, recordsize 8k, zvol: Python 2.7.13.- pve-ubuntu-fnas-1: Disk Scheduler: DEADLINE. Python 2.7.12.

ZFS SSD Benchmark: RAW IMAGE vs QCOW2 vs ZVOL for KVMaio-stress: Rand Writecompilebench: Compilecompilebench: Initial Createcompilebench: Read Compiled Treedbench: 1dbench: 6dbench: 12dbench: 48dbench: 128dbench: 256iozone: 1MB - 8GB - Read Performanceiozone: 4Kb - 8GB - Read Performanceiozone: 1MB - 8GB - Write Performanceiozone: 4Kb - 8GB - Write Performanceiozone: 64Kb - 8GB - Read Performanceiozone: 64Kb - 8GB - Write Performancezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvolpve-ubuntu-fnas-1125.74142.8882.4064.6942.33102.98162.59294.03203.76118.70215.34223.31156.30161.59225.08156.04131.81147.1886.8564.8342.60105.25166.12292.22207.32120.18216.63224.74157.79162.04226.37160.28111.3381.5950.1160.7242.7288.54133.55177.89102.0762.0086.0794.9966.7963.1088.9060.0393.7061.6542.5456.7242.2089.99137.50176.4189.7555.57165.90173.6865.2261.43174.5658.8798.0172.1348.4055.7242.1891.96137.66181.3897.7955.77167.07174.6360.0057.00172.3855.72100.8791.6949.3246.5840.8786.79128.76153.6388.4158.4780.5084.9572.0774.0684.8667.621449.77117.05100.8964.91179.20320.86387.51437.60259.95333.19111.88112.02111.94111.83112.04111.94OpenBenchmarking.org

AIO-Stress

Test: Random Write

OpenBenchmarking.orgMB/s, More Is BetterAIO-Stress 0.21Test: Random Writepve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 image30060090012001500SE +/- 21.95, N = 5SE +/- 0.94, N = 3SE +/- 0.78, N = 3SE +/- 0.51, N = 3SE +/- 3.01, N = 6SE +/- 0.50, N = 3SE +/- 1.19, N = 31449.77131.81125.74111.33100.8798.0193.701. (CC) gcc options: -pthread -laio

Compile Bench

Test: Compile

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Compilezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagepve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 image306090120150SE +/- 5.21, N = 6SE +/- 5.01, N = 6SE +/- 0.37, N = 3SE +/- 2.73, N = 6SE +/- 1.40, N = 3SE +/- 2.12, N = 6SE +/- 4.97, N = 6147.18142.88117.0591.6981.5972.1361.65

Compile Bench

Test: Initial Create

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Initial Createpve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 image20406080100SE +/- 2.22, N = 3SE +/- 3.08, N = 3SE +/- 6.42, N = 3SE +/- 5.96, N = 3SE +/- 3.57, N = 3SE +/- 1.91, N = 3SE +/- 8.32, N = 3100.8986.8582.4050.1149.3248.4042.54

Compile Bench

Test: Read Compiled Tree

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Read Compiled Treepve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol1428425670SE +/- 0.23, N = 3SE +/- 0.33, N = 3SE +/- 0.14, N = 3SE +/- 2.51, N = 3SE +/- 0.52, N = 3SE +/- 0.15, N = 3SE +/- 0.65, N = 364.9164.8364.6960.7256.7255.7246.58

Dbench

Client Count: 1

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 1pve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol4080120160200SE +/- 0.06, N = 3SE +/- 0.01, N = 3SE +/- 0.09, N = 3SE +/- 0.08, N = 3SE +/- 0.02, N = 3SE +/- 0.05, N = 3SE +/- 0.15, N = 3179.2042.7242.6042.3342.2042.1840.871. (CC) gcc options: -lpopt -O2

Dbench

Client Count: 6

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 6pve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, zvol70140210280350SE +/- 6.43, N = 6SE +/- 1.07, N = 3SE +/- 0.91, N = 3SE +/- 1.36, N = 3SE +/- 0.90, N = 3SE +/- 1.27, N = 3SE +/- 0.16, N = 3320.86105.25102.9891.9689.9988.5486.791. (CC) gcc options: -lpopt -O2

Dbench

Client Count: 12

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 12pve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, zvol80160240320400SE +/- 2.88, N = 3SE +/- 0.58, N = 3SE +/- 0.75, N = 3SE +/- 1.52, N = 3SE +/- 1.47, N = 3SE +/- 0.41, N = 3SE +/- 0.84, N = 3387.51166.12162.59137.66137.50133.55128.761. (CC) gcc options: -lpopt -O2

Dbench

Client Count: 48

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 48pve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvol90180270360450SE +/- 4.55, N = 3SE +/- 4.38, N = 3SE +/- 4.21, N = 3SE +/- 7.08, N = 6SE +/- 6.43, N = 6SE +/- 7.08, N = 6SE +/- 6.84, N = 6437.60294.03292.22181.38177.89176.41153.631. (CC) gcc options: -lpopt -O2

Dbench

Client Count: 128

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 128pve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvol60120180240300SE +/- 6.29, N = 6SE +/- 2.00, N = 3SE +/- 0.99, N = 3SE +/- 4.96, N = 6SE +/- 5.85, N = 6SE +/- 5.76, N = 6SE +/- 4.52, N = 6259.95207.32203.76102.0797.7989.7588.411. (CC) gcc options: -lpopt -O2

Dbench

Client Count: 256

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 256pve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 image70140210280350SE +/- 6.69, N = 3SE +/- 0.40, N = 3SE +/- 0.23, N = 3SE +/- 2.00, N = 6SE +/- 1.39, N = 6SE +/- 2.11, N = 6SE +/- 0.85, N = 3333.19120.18118.7062.0058.4755.7755.571. (CC) gcc options: -lpopt -O2

Flexible IO Tester

Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 image100200300400500SE +/- 7.46, N = 6SE +/- 10.34, N = 6SE +/- 2.08, N = 3SE +/- 1.20, N = 3SE +/- 1.00, N = 3SE +/- 1.00, N = 34734542942922302281. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw image918273645SE +/- 0.52, N = 3SE +/- 0.65, N = 4SE +/- 0.38, N = 3SE +/- 0.03, N = 3SE +/- 0.10, N = 3SE +/- 0.40, N = 340.7340.6539.2038.9338.6036.801. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, raw image50100150200250SE +/- 1.45, N = 3SE +/- 2.52, N = 3SE +/- 0.88, N = 3SE +/- 0.58, N = 32392372322302302291. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, zvol306090120150SE +/- 0.58, N = 3SE +/- 0.67, N = 3SE +/- 0.31, N = 3SE +/- 0.09, N = 3SE +/- 2.09, N = 6SE +/- 1.08, N = 6146.00143.0071.2070.2354.7542.051. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, zvol714212835SE +/- 0.20, N = 3SE +/- 0.19, N = 3SE +/- 0.10, N = 3SE +/- 0.26, N = 3SE +/- 0.24, N = 3SE +/- 0.21, N = 429.2025.7320.7017.2015.9712.051. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvol306090120150SE +/- 0.88, N = 3SE +/- 0.67, N = 3SE +/- 0.81, N = 3SE +/- 0.03, N = 3SE +/- 0.25, N = 3SE +/- 3.30, N = 6118.00113.0066.4061.3359.4038.751. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 image100200300400500SE +/- 10.48, N = 6SE +/- 5.61, N = 3SE +/- 1.20, N = 3SE +/- 0.67, N = 3SE +/- 0.67, N = 34724582912892292261. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvol1326395265SE +/- 2.11, N = 6SE +/- 0.23, N = 3SE +/- 1.33, N = 6SE +/- 1.52, N = 6SE +/- 1.10, N = 6SE +/- 0.32, N = 358.3255.5054.1745.8545.0244.731. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 image80160240320400SE +/- 4.51, N = 3SE +/- 7.51, N = 6SE +/- 1.33, N = 3SE +/- 1.76, N = 3SE +/- 1.33, N = 3SE +/- 0.58, N = 33533473063032522441. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, zvol306090120150SE +/- 2.33, N = 3SE +/- 0.93, N = 3SE +/- 1.08, N = 3SE +/- 1.94, N = 6SE +/- 6.42, N = 6152.00147.0074.3772.0366.2845.331. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, zvol1122334455SE +/- 1.80, N = 6SE +/- 1.72, N = 6SE +/- 1.78, N = 6SE +/- 1.80, N = 6SE +/- 0.64, N = 6SE +/- 0.57, N = 647.1046.3041.8332.7029.7724.151. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, zvol306090120150SE +/- 0.67, N = 3SE +/- 0.41, N = 3SE +/- 0.59, N = 3SE +/- 0.03, N = 3SE +/- 3.31, N = 6139.00138.0074.2771.3770.7736.431. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

IOzone

Record Size: 1MB - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 8GB - Disk Test: Read Performancezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagepve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, zvol50100150200250SE +/- 0.57, N = 3SE +/- 0.79, N = 3SE +/- 1.25, N = 3SE +/- 1.22, N = 3SE +/- 0.24, N = 3SE +/- 0.58, N = 3SE +/- 1.26, N = 3216.63215.34167.07165.90111.8886.0780.501. (CC) gcc options: -O3

IOzone

Record Size: 4Kb - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 8GB - Disk Test: Read Performancezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagepve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, zvol50100150200250SE +/- 2.04, N = 3SE +/- 0.88, N = 3SE +/- 0.79, N = 3SE +/- 0.12, N = 3SE +/- 0.09, N = 3SE +/- 1.27, N = 3SE +/- 2.19, N = 6224.74223.31174.63173.68112.0294.9984.951. (CC) gcc options: -O3

IOzone

Record Size: 1MB - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 8GB - Disk Test: Write Performancezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagepve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw image306090120150SE +/- 0.85, N = 3SE +/- 0.82, N = 3SE +/- 0.04, N = 3SE +/- 2.02, N = 6SE +/- 0.50, N = 3SE +/- 2.01, N = 6SE +/- 1.42, N = 6157.79156.30111.9472.0766.7965.2260.001. (CC) gcc options: -O3

IOzone

Record Size: 4Kb - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 8GB - Disk Test: Write Performancezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagepve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw image4080120160200SE +/- 1.03, N = 3SE +/- 1.79, N = 3SE +/- 0.03, N = 3SE +/- 1.13, N = 3SE +/- 1.08, N = 3SE +/- 1.02, N = 6SE +/- 1.60, N = 6162.04161.59111.8374.0663.1061.4357.001. (CC) gcc options: -O3

IOzone

Record Size: 64Kb - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 8GB - Disk Test: Read Performancezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagepve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, zvol50100150200250SE +/- 0.36, N = 3SE +/- 0.72, N = 3SE +/- 0.49, N = 3SE +/- 0.49, N = 3SE +/- 0.08, N = 3SE +/- 1.37, N = 4SE +/- 2.08, N = 6226.37225.08174.56172.38112.0488.9084.861. (CC) gcc options: -O3

IOzone

Record Size: 64Kb - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 8GB - Disk Test: Write Performancezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagepve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw image4080120160200SE +/- 2.94, N = 6SE +/- 2.73, N = 6SE +/- 0.04, N = 3SE +/- 3.14, N = 6SE +/- 1.77, N = 6SE +/- 0.92, N = 6SE +/- 0.37, N = 3160.28156.04111.9467.6260.0358.8755.721. (CC) gcc options: -O3

Phoronix Test Suite v10.8.4