ZFS SSD Benchmark: RAW IMAGE vs QCOW2 vs ZVOL for KVM

FNAS

HTML result view exported from: https://openbenchmarking.org/result/1803243-KH-1803231TY25&sro&grt.

ZFS SSD Benchmark: RAW IMAGE vs QCOW2 vs ZVOL for KVMProcessorMotherboardChipsetMemoryDiskGraphicsNetworkOSKernelCompilerFile-SystemScreen ResolutionSystem Layerzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvolpve-ubuntu-fnas-18 x Intel Celeron_4x0 (Conroe/Merom Class Core 2) @ 1.86GHz (8 Cores)QEMU Standard PC (i440FX + PIIX 1996)Intel 440FX- 82441FX PMC1 x 1024 MB RAM QEMU54GB QEMU HDDCirrus Logic GD 5446Realtek RTL-8100/8101L/8139Debian 9.14.9.0-4-amd64 (x86_64)GCC 6.3.0 20170516ext41024x768qemuCommon KVM @ 3.09GHz (2 Cores)2048MB537GB QEMU HDDDevice 1234:1111Red Hat Virtio deviceUbuntu 16.044.4.0-116-generic (x86_64)GCC 5.4.0 20160609800x600OpenBenchmarking.orgCompiler Details- zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 64k, raw image: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 64k, zvol: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 8k, raw image: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 8k, zvol: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - pve-ubuntu-fnas-1: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --disable-werror --enable-checking=release --enable-clocale=gnu --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-tune=generic -v Disk Details- zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image: CFQ / data=ordered,errors=remount-ro,relatime,rw- zfs ssd mirror, ashift=13, recordsize 64k, raw image: CFQ / data=ordered,errors=remount-ro,relatime,rw- zfs ssd mirror, ashift=13, recordsize 64k, zvol: CFQ / data=ordered,errors=remount-ro,relatime,rw- zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image: CFQ / data=ordered,errors=remount-ro,relatime,rw- zfs ssd mirror, ashift=13, recordsize 8k, raw image: CFQ / data=ordered,errors=remount-ro,relatime,rw- zfs ssd mirror, ashift=13, recordsize 8k, zvol: CFQ / data=ordered,errors=remount-ro,relatime,rw- pve-ubuntu-fnas-1: DEADLINE / data=ordered,errors=remount-ro,relatime,rwSystem Details- zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image: Python 2.7.13.- zfs ssd mirror, ashift=13, recordsize 64k, raw image: Python 2.7.13.- zfs ssd mirror, ashift=13, recordsize 64k, zvol: Python 2.7.13.- zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image: Python 2.7.13.- zfs ssd mirror, ashift=13, recordsize 8k, raw image: Python 2.7.13.- zfs ssd mirror, ashift=13, recordsize 8k, zvol: Python 2.7.13.- pve-ubuntu-fnas-1: Disk Scheduler: DEADLINE. Python 2.7.12.

ZFS SSD Benchmark: RAW IMAGE vs QCOW2 vs ZVOL for KVMaio-stress: Rand Writecompilebench: Compilecompilebench: Initial Createcompilebench: Read Compiled Treedbench: 1dbench: 6dbench: 12dbench: 48dbench: 128dbench: 256iozone: 1MB - 8GB - Read Performanceiozone: 4Kb - 8GB - Read Performanceiozone: 1MB - 8GB - Write Performanceiozone: 4Kb - 8GB - Write Performanceiozone: 64Kb - 8GB - Read Performanceiozone: 64Kb - 8GB - Write Performancezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvolpve-ubuntu-fnas-1125.74142.8882.4064.6942.33102.98162.59294.03203.76118.70215.34223.31156.30161.59225.08156.04131.81147.1886.8564.8342.60105.25166.12292.22207.32120.18216.63224.74157.79162.04226.37160.28111.3381.5950.1160.7242.7288.54133.55177.89102.0762.0086.0794.9966.7963.1088.9060.0393.7061.6542.5456.7242.2089.99137.50176.4189.7555.57165.90173.6865.2261.43174.5658.8798.0172.1348.4055.7242.1891.96137.66181.3897.7955.77167.07174.6360.0057.00172.3855.72100.8791.6949.3246.5840.8786.79128.76153.6388.4158.4780.5084.9572.0774.0684.8667.621449.77117.05100.8964.91179.20320.86387.51437.60259.95333.19111.88112.02111.94111.83112.04111.94OpenBenchmarking.org

AIO-Stress

Test: Random Write

OpenBenchmarking.orgMB/s, More Is BetterAIO-Stress 0.21Test: Random Writepve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol30060090012001500SE +/- 21.95, N = 5SE +/- 0.78, N = 3SE +/- 0.94, N = 3SE +/- 0.51, N = 3SE +/- 1.19, N = 3SE +/- 0.50, N = 3SE +/- 3.01, N = 61449.77125.74131.81111.3393.7098.01100.871. (CC) gcc options: -pthread -laio

Compile Bench

Test: Compile

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Compilepve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol306090120150SE +/- 0.37, N = 3SE +/- 5.01, N = 6SE +/- 5.21, N = 6SE +/- 1.40, N = 3SE +/- 4.97, N = 6SE +/- 2.12, N = 6SE +/- 2.73, N = 6117.05142.88147.1881.5961.6572.1391.69

Compile Bench

Test: Initial Create

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Initial Createpve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol20406080100SE +/- 2.22, N = 3SE +/- 6.42, N = 3SE +/- 3.08, N = 3SE +/- 5.96, N = 3SE +/- 8.32, N = 3SE +/- 1.91, N = 3SE +/- 3.57, N = 3100.8982.4086.8550.1142.5448.4049.32

Compile Bench

Test: Read Compiled Tree

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Read Compiled Treepve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol1428425670SE +/- 0.23, N = 3SE +/- 0.14, N = 3SE +/- 0.33, N = 3SE +/- 2.51, N = 3SE +/- 0.52, N = 3SE +/- 0.15, N = 3SE +/- 0.65, N = 364.9164.6964.8360.7256.7255.7246.58

Dbench

Client Count: 1

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 1pve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol4080120160200SE +/- 0.06, N = 3SE +/- 0.08, N = 3SE +/- 0.09, N = 3SE +/- 0.01, N = 3SE +/- 0.02, N = 3SE +/- 0.05, N = 3SE +/- 0.15, N = 3179.2042.3342.6042.7242.2042.1840.871. (CC) gcc options: -lpopt -O2

Dbench

Client Count: 6

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 6pve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol70140210280350SE +/- 6.43, N = 6SE +/- 0.91, N = 3SE +/- 1.07, N = 3SE +/- 1.27, N = 3SE +/- 0.90, N = 3SE +/- 1.36, N = 3SE +/- 0.16, N = 3320.86102.98105.2588.5489.9991.9686.791. (CC) gcc options: -lpopt -O2

Dbench

Client Count: 12

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 12pve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol80160240320400SE +/- 2.88, N = 3SE +/- 0.75, N = 3SE +/- 0.58, N = 3SE +/- 0.41, N = 3SE +/- 1.47, N = 3SE +/- 1.52, N = 3SE +/- 0.84, N = 3387.51162.59166.12133.55137.50137.66128.761. (CC) gcc options: -lpopt -O2

Dbench

Client Count: 48

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 48pve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol90180270360450SE +/- 4.55, N = 3SE +/- 4.38, N = 3SE +/- 4.21, N = 3SE +/- 6.43, N = 6SE +/- 7.08, N = 6SE +/- 7.08, N = 6SE +/- 6.84, N = 6437.60294.03292.22177.89176.41181.38153.631. (CC) gcc options: -lpopt -O2

Dbench

Client Count: 128

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 128pve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol60120180240300SE +/- 6.29, N = 6SE +/- 0.99, N = 3SE +/- 2.00, N = 3SE +/- 4.96, N = 6SE +/- 5.76, N = 6SE +/- 5.85, N = 6SE +/- 4.52, N = 6259.95203.76207.32102.0789.7597.7988.411. (CC) gcc options: -lpopt -O2

Dbench

Client Count: 256

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 256pve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol70140210280350SE +/- 6.69, N = 3SE +/- 0.23, N = 3SE +/- 0.40, N = 3SE +/- 2.00, N = 6SE +/- 0.85, N = 3SE +/- 2.11, N = 6SE +/- 1.39, N = 6333.19118.70120.1862.0055.5755.7758.471. (CC) gcc options: -lpopt -O2

Flexible IO Tester

Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol100200300400500SE +/- 1.20, N = 3SE +/- 2.08, N = 3SE +/- 10.34, N = 6SE +/- 1.00, N = 3SE +/- 1.00, N = 3SE +/- 7.46, N = 62922944542282304731. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol918273645SE +/- 0.10, N = 3SE +/- 0.03, N = 3SE +/- 0.52, N = 3SE +/- 0.65, N = 4SE +/- 0.40, N = 3SE +/- 0.38, N = 338.6038.9340.7340.6536.8039.201. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol50100150200250SE +/- 1.45, N = 3SE +/- 0.58, N = 3SE +/- 2.52, N = 3SE +/- 0.88, N = 32372392302322292301. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol306090120150SE +/- 0.67, N = 3SE +/- 0.58, N = 3SE +/- 2.09, N = 6SE +/- 0.09, N = 3SE +/- 0.31, N = 3SE +/- 1.08, N = 6143.00146.0054.7570.2371.2042.051. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol714212835SE +/- 0.24, N = 3SE +/- 0.26, N = 3SE +/- 0.21, N = 4SE +/- 0.19, N = 3SE +/- 0.20, N = 3SE +/- 0.10, N = 315.9717.2012.0525.7329.2020.701. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol306090120150SE +/- 0.67, N = 3SE +/- 0.88, N = 3SE +/- 0.81, N = 3SE +/- 0.25, N = 3SE +/- 0.03, N = 3SE +/- 3.30, N = 6113.00118.0066.4059.4061.3338.751. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol100200300400500SE +/- 1.20, N = 3SE +/- 0.67, N = 3SE +/- 5.61, N = 3SE +/- 0.67, N = 3SE +/- 10.48, N = 62912894582262294721. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol1326395265SE +/- 1.33, N = 6SE +/- 0.23, N = 3SE +/- 0.32, N = 3SE +/- 1.52, N = 6SE +/- 2.11, N = 6SE +/- 1.10, N = 654.1755.5044.7345.8558.3245.021. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol80160240320400SE +/- 1.33, N = 3SE +/- 1.76, N = 3SE +/- 7.51, N = 6SE +/- 0.58, N = 3SE +/- 1.33, N = 3SE +/- 4.51, N = 33063033472442523531. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol306090120150SE +/- 2.33, N = 3SE +/- 1.94, N = 6SE +/- 1.08, N = 3SE +/- 0.93, N = 3SE +/- 6.42, N = 6147.00152.0066.2872.0374.3745.331. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol1122334455SE +/- 1.72, N = 6SE +/- 1.80, N = 6SE +/- 0.64, N = 6SE +/- 1.80, N = 6SE +/- 1.78, N = 6SE +/- 0.57, N = 646.3047.1029.7732.7041.8324.151. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol306090120150SE +/- 0.67, N = 3SE +/- 0.59, N = 3SE +/- 0.03, N = 3SE +/- 0.41, N = 3SE +/- 3.31, N = 6139.00138.0071.3770.7774.2736.431. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

IOzone

Record Size: 1MB - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 8GB - Disk Test: Read Performancepve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol50100150200250SE +/- 0.24, N = 3SE +/- 0.79, N = 3SE +/- 0.57, N = 3SE +/- 0.58, N = 3SE +/- 1.22, N = 3SE +/- 1.25, N = 3SE +/- 1.26, N = 3111.88215.34216.6386.07165.90167.0780.501. (CC) gcc options: -O3

IOzone

Record Size: 4Kb - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 8GB - Disk Test: Read Performancepve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol50100150200250SE +/- 0.09, N = 3SE +/- 0.88, N = 3SE +/- 2.04, N = 3SE +/- 1.27, N = 3SE +/- 0.12, N = 3SE +/- 0.79, N = 3SE +/- 2.19, N = 6112.02223.31224.7494.99173.68174.6384.951. (CC) gcc options: -O3

IOzone

Record Size: 1MB - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 8GB - Disk Test: Write Performancepve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol306090120150SE +/- 0.04, N = 3SE +/- 0.82, N = 3SE +/- 0.85, N = 3SE +/- 0.50, N = 3SE +/- 2.01, N = 6SE +/- 1.42, N = 6SE +/- 2.02, N = 6111.94156.30157.7966.7965.2260.0072.071. (CC) gcc options: -O3

IOzone

Record Size: 4Kb - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 8GB - Disk Test: Write Performancepve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol4080120160200SE +/- 0.03, N = 3SE +/- 1.79, N = 3SE +/- 1.03, N = 3SE +/- 1.08, N = 3SE +/- 1.02, N = 6SE +/- 1.60, N = 6SE +/- 1.13, N = 3111.83161.59162.0463.1061.4357.0074.061. (CC) gcc options: -O3

IOzone

Record Size: 64Kb - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 8GB - Disk Test: Read Performancepve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol50100150200250SE +/- 0.08, N = 3SE +/- 0.72, N = 3SE +/- 0.36, N = 3SE +/- 1.37, N = 4SE +/- 0.49, N = 3SE +/- 0.49, N = 3SE +/- 2.08, N = 6112.04225.08226.3788.90174.56172.3884.861. (CC) gcc options: -O3

IOzone

Record Size: 64Kb - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 8GB - Disk Test: Write Performancepve-ubuntu-fnas-1zfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvol4080120160200SE +/- 0.04, N = 3SE +/- 2.73, N = 6SE +/- 2.94, N = 6SE +/- 1.77, N = 6SE +/- 0.92, N = 6SE +/- 0.37, N = 3SE +/- 3.14, N = 6111.94156.04160.2860.0358.8755.7267.621. (CC) gcc options: -O3

Phoronix Test Suite v10.8.4