ZFS SSD Benchmark: RAW IMAGE vs QCOW2 vs ZVOL for KVM

This benchmark show’s the performance of a zfs pool
providing storage to a kvm virtual machine with three different formats:

raw image files on plain dataset
qcow2 image file on plain dataset
zvol

For each of those several filesystem benchmarks from the phoronix-benchmark-suite are run with two different recordsizes:

8k
64k

Host Details:

cpu: Intel® Xeon® CPU E5320 @ 1.86GHz
ram: 6GB DDR2
sata: Intel Corporation 631xESB/632xESB SATA AHCI Controller (rev 09
os: Debian GNU/Linux 9.4 (stretch), fully upgraded as of 2018-03-15
zfs: 0.7.6-1~bpo9+1 (from stretch-backports)
ssd: 2x samsung 840 pro 256GB

KVM machine:

cpu: host
ram: 1024MB
disk: bus=scsi,cache=writeback,io=threads
os: Debian GNU/Linux 9.1 (stretch)
fs: ext4

The zfs pool used consists of a single mirrored vdev with samsung 840 pro ssd’s.
Each benchmark is run like this:

drop all caches
trim whole ssd’s
create zfs pool (ashift=13,compression=lz4,atime=on,relatime=on
benchmark,devices=off,exec=off,setuid=off,xattr=off)
create disk:

qemu-img create -f raw debian9.raw 50G
qemu-img create -f qcow2 -o cluster_size=8k,preallocation=metadata,compat=1.1,lazy_refcounts=on debian9.qcow2 50G
zfs create -o volblocksize=8k -V 50G benchmark/kvm/debian9


create kvm machine
take timestamp
let debian9 install automatically
save install time
install phoronix-test-suite and needed dependencies in the vm
run the pts
trim the filesystem and save the time it took
copy the results to the kvm host
destroy vm and zpool

other test results:

8k_raw - install time: 619s, trim: 39.401s
64k raw - install time: 589s, trim: 0m9.593s
8k qcow2 - install time: 615s, trim: 1m45.421s
64k qcow2 - install time: 590s, trim: 0m7.598s
8k zvol - install time: 621s, trim: 0m11.353s
64k zvol - install time: 605s

Compare your own system(s) to this result file with the Phoronix Test Suite by running the command: phoronix-test-suite benchmark 1803243-KH-1803231TY84
Jump To Table - Results

View

Do Not Show Noisy Results
Do Not Show Results With Incomplete Data
Do Not Show Results With Little Change/Spread
List Notable Results
Show Result Confidence Charts

Statistics

Show Overall Harmonic Mean(s)
Show Overall Geometric Mean
Show Wins / Losses Counts (Pie Chart)
Normalize Results
Remove Outliers Before Calculating Averages

Graph Settings

Force Line Graphs Where Applicable
Convert To Scalar Where Applicable
Prefer Vertical Bar Graphs

Multi-Way Comparison

Condense Multi-Option Tests Into Single Result Graphs

Table

Show Detailed System Result Table

Run Management

Highlight
Result
Hide
Result
Result
Identifier
Performance Per
Dollar
Date
Run
  Test
  Duration
zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image
March 17 2018
 
zfs ssd mirror, ashift=13, recordsize 64k, raw image
March 16 2018
 
zfs ssd mirror, ashift=13, recordsize 64k, zvol
March 18 2018
 
zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image
March 16 2018
 
zfs ssd mirror, ashift=13, recordsize 8k, raw image
March 15 2018
 
zfs ssd mirror, ashift=13, recordsize 8k, zvol
March 17 2018
 
pve-ubuntu-local-1
March 24 2018
 
Invert Hiding All Results Option
 

Only show results where is faster than
Only show results matching title/arguments (delimit multiple options with a comma):
Do not show results matching title/arguments (delimit multiple options with a comma):


ZFS SSD Benchmark: RAW IMAGE vs QCOW2 vs ZVOL for KVMProcessorMotherboardChipsetMemoryDiskGraphicsNetworkOSKernelCompilerFile-SystemScreen ResolutionSystem Layerzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvolpve-ubuntu-local-18 x Intel Celeron_4x0 (Conroe/Merom Class Core 2) @ 1.86GHz (8 Cores)QEMU Standard PC (i440FX + PIIX 1996)Intel 440FX- 82441FX PMC1 x 1024 MB RAM QEMU54GB QEMU HDDCirrus Logic GD 5446Realtek RTL-8100/8101L/8139Debian 9.14.9.0-4-amd64 (x86_64)GCC 6.3.0 20170516ext41024x768qemuCommon KVM @ 3.09GHz (2 Cores)2048MB34GB QEMU HDDDevice 1234:1111Red Hat Virtio deviceUbuntu 16.044.4.0-116-generic (x86_64)GCC 5.4.0 20160609800x600OpenBenchmarking.orgCompiler Details- zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 64k, raw image: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 64k, zvol: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 8k, raw image: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 8k, zvol: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - pve-ubuntu-local-1: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --disable-werror --enable-checking=release --enable-clocale=gnu --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-tune=generic -v Disk Details- zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image: CFQ / data=ordered,errors=remount-ro,relatime,rw- zfs ssd mirror, ashift=13, recordsize 64k, raw image: CFQ / data=ordered,errors=remount-ro,relatime,rw- zfs ssd mirror, ashift=13, recordsize 64k, zvol: CFQ / data=ordered,errors=remount-ro,relatime,rw- zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image: CFQ / data=ordered,errors=remount-ro,relatime,rw- zfs ssd mirror, ashift=13, recordsize 8k, raw image: CFQ / data=ordered,errors=remount-ro,relatime,rw- zfs ssd mirror, ashift=13, recordsize 8k, zvol: CFQ / data=ordered,errors=remount-ro,relatime,rw- pve-ubuntu-local-1: DEADLINE / data=ordered,errors=remount-ro,relatime,rwSystem Details- zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image: Python 2.7.13.- zfs ssd mirror, ashift=13, recordsize 64k, raw image: Python 2.7.13.- zfs ssd mirror, ashift=13, recordsize 64k, zvol: Python 2.7.13.- zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image: Python 2.7.13.- zfs ssd mirror, ashift=13, recordsize 8k, raw image: Python 2.7.13.- zfs ssd mirror, ashift=13, recordsize 8k, zvol: Python 2.7.13.- pve-ubuntu-local-1: Disk Scheduler: DEADLINE. Python 2.7.12.

zfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvolpve-ubuntu-local-1Result OverviewPhoronix Test Suite100%220%339%459%579%DbenchCompile BenchIOzoneAIO-Stress

ZFS SSD Benchmark: RAW IMAGE vs QCOW2 vs ZVOL for KVMaio-stress: Rand Writecompilebench: Compilecompilebench: Initial Createcompilebench: Read Compiled Treedbench: 1dbench: 6dbench: 12dbench: 48dbench: 128dbench: 256iozone: 1MB - 8GB - Read Performanceiozone: 4Kb - 8GB - Read Performanceiozone: 1MB - 8GB - Write Performanceiozone: 4Kb - 8GB - Write Performanceiozone: 64Kb - 8GB - Read Performanceiozone: 64Kb - 8GB - Write Performancezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvolpve-ubuntu-local-1125.74142.8882.4064.6942.33102.98162.59294.03203.76118.70215.34223.31156.30161.59225.08156.04131.81147.1886.8564.8342.60105.25166.12292.22207.32120.18216.63224.74157.79162.04226.37160.28111.3381.5950.1160.7242.7288.54133.55177.89102.0762.0086.0794.9966.7963.1088.9060.0393.7061.6542.5456.7242.2089.99137.50176.4189.7555.57165.90173.6865.2261.43174.5658.8798.0172.1348.4055.7242.1891.96137.66181.3897.7955.77167.07174.6360.0057.00172.3855.72100.8791.6949.3246.5840.8786.79128.76153.6388.4158.4780.5084.9572.0774.0684.8667.6249.15262.57182.82166.58430.74826.51779.05792.05308.19201.67245.63256.87206.19207.63262.94207.41OpenBenchmarking.org

AIO-Stress

AIO-Stress is an a-synchronous I/O benchmark created by SuSE. Current this profile uses a 2048MB test file and a 64KB record size. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgMB/s, More Is BetterAIO-Stress 0.21Test: Random Writepve-ubuntu-local-1zfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw image306090120150SE +/- 0.83, N = 4SE +/- 1.19, N = 3SE +/- 0.50, N = 3SE +/- 3.01, N = 6SE +/- 0.51, N = 3SE +/- 0.78, N = 3SE +/- 0.94, N = 349.1593.7098.01100.87111.33125.74131.811. (CC) gcc options: -pthread -laio

Compile Bench

Compilebench tries to age a filesystem by simulating some of the disk IO common in creating, compiling, patching, stating and reading kernel trees. It indirectly measures how well filesystems can maintain directory locality as the disk fills up and directories age. This current test is setup to use the makej mode with 10 initial directories Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Compilezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagepve-ubuntu-local-160120180240300SE +/- 4.97, N = 6SE +/- 2.12, N = 6SE +/- 1.40, N = 3SE +/- 2.73, N = 6SE +/- 5.01, N = 6SE +/- 5.21, N = 6SE +/- 4.39, N = 661.6572.1381.5991.69142.88147.18262.57

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Initial Createzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagepve-ubuntu-local-14080120160200SE +/- 8.32, N = 3SE +/- 1.91, N = 3SE +/- 3.57, N = 3SE +/- 5.96, N = 3SE +/- 6.42, N = 3SE +/- 3.08, N = 3SE +/- 10.19, N = 342.5448.4049.3250.1182.4086.85182.82

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Read Compiled Treezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagepve-ubuntu-local-14080120160200SE +/- 0.65, N = 3SE +/- 0.15, N = 3SE +/- 0.52, N = 3SE +/- 2.51, N = 3SE +/- 0.14, N = 3SE +/- 0.33, N = 3SE +/- 2.00, N = 346.5855.7256.7260.7264.6964.83166.58

Dbench

Dbench is a benchmark designed by the Samba project as a free alternative to netbench, but dbench contains only file-system calls for testing the disk performance. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 1zfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolpve-ubuntu-local-190180270360450SE +/- 0.15, N = 3SE +/- 0.05, N = 3SE +/- 0.02, N = 3SE +/- 0.08, N = 3SE +/- 0.09, N = 3SE +/- 0.01, N = 3SE +/- 3.32, N = 340.8742.1842.2042.3342.6042.72430.741. (CC) gcc options: -lpopt -O2

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 6zfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagepve-ubuntu-local-12004006008001000SE +/- 0.16, N = 3SE +/- 1.27, N = 3SE +/- 0.90, N = 3SE +/- 1.36, N = 3SE +/- 0.91, N = 3SE +/- 1.07, N = 3SE +/- 15.67, N = 386.7988.5489.9991.96102.98105.25826.511. (CC) gcc options: -lpopt -O2

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 12zfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagepve-ubuntu-local-12004006008001000SE +/- 0.84, N = 3SE +/- 0.41, N = 3SE +/- 1.47, N = 3SE +/- 1.52, N = 3SE +/- 0.75, N = 3SE +/- 0.58, N = 3SE +/- 9.36, N = 3128.76133.55137.50137.66162.59166.12779.051. (CC) gcc options: -lpopt -O2

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 48zfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagepve-ubuntu-local-12004006008001000SE +/- 6.84, N = 6SE +/- 7.08, N = 6SE +/- 6.43, N = 6SE +/- 7.08, N = 6SE +/- 4.21, N = 3SE +/- 4.38, N = 3SE +/- 2.73, N = 3153.63176.41177.89181.38292.22294.03792.051. (CC) gcc options: -lpopt -O2

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 128zfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagepve-ubuntu-local-170140210280350SE +/- 4.52, N = 6SE +/- 5.76, N = 6SE +/- 5.85, N = 6SE +/- 4.96, N = 6SE +/- 0.99, N = 3SE +/- 2.00, N = 3SE +/- 6.16, N = 388.4189.7597.79102.07203.76207.32308.191. (CC) gcc options: -lpopt -O2

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 256zfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagepve-ubuntu-local-14080120160200SE +/- 0.85, N = 3SE +/- 2.11, N = 6SE +/- 1.39, N = 6SE +/- 2.00, N = 6SE +/- 0.23, N = 3SE +/- 0.40, N = 3SE +/- 1.90, N = 355.5755.7758.4762.00118.70120.18201.671. (CC) gcc options: -lpopt -O2

Flexible IO Tester

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, zvol100200300400500SE +/- 1.00, N = 3SE +/- 1.00, N = 3SE +/- 1.20, N = 3SE +/- 2.08, N = 3SE +/- 10.34, N = 6SE +/- 7.46, N = 62282302922944544731. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, zvol918273645SE +/- 0.40, N = 3SE +/- 0.10, N = 3SE +/- 0.03, N = 3SE +/- 0.38, N = 3SE +/- 0.65, N = 4SE +/- 0.52, N = 336.8038.6038.9339.2040.6540.731. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw image50100150200250SE +/- 0.58, N = 3SE +/- 0.88, N = 3SE +/- 2.52, N = 3SE +/- 1.45, N = 32292302302322372391. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw image306090120150SE +/- 1.08, N = 6SE +/- 2.09, N = 6SE +/- 0.09, N = 3SE +/- 0.31, N = 3SE +/- 0.67, N = 3SE +/- 0.58, N = 342.0554.7570.2371.20143.00146.001. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw image714212835SE +/- 0.21, N = 4SE +/- 0.24, N = 3SE +/- 0.26, N = 3SE +/- 0.10, N = 3SE +/- 0.19, N = 3SE +/- 0.20, N = 312.0515.9717.2020.7025.7329.201. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw image306090120150SE +/- 3.30, N = 6SE +/- 0.25, N = 3SE +/- 0.03, N = 3SE +/- 0.81, N = 3SE +/- 0.67, N = 3SE +/- 0.88, N = 338.7559.4061.3366.40113.00118.001. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, zvol100200300400500SE +/- 0.67, N = 3SE +/- 0.67, N = 3SE +/- 1.20, N = 3SE +/- 5.61, N = 3SE +/- 10.48, N = 62262292892914584721. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, raw image1326395265SE +/- 0.32, N = 3SE +/- 1.10, N = 6SE +/- 1.52, N = 6SE +/- 1.33, N = 6SE +/- 0.23, N = 3SE +/- 2.11, N = 644.7345.0245.8554.1755.5058.321. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, zvol80160240320400SE +/- 0.58, N = 3SE +/- 1.33, N = 3SE +/- 1.76, N = 3SE +/- 1.33, N = 3SE +/- 7.51, N = 6SE +/- 4.51, N = 32442523033063473531. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw image306090120150SE +/- 6.42, N = 6SE +/- 1.94, N = 6SE +/- 1.08, N = 3SE +/- 0.93, N = 3SE +/- 2.33, N = 345.3366.2872.0374.37147.00152.001. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw image1122334455SE +/- 0.57, N = 6SE +/- 0.64, N = 6SE +/- 1.80, N = 6SE +/- 1.78, N = 6SE +/- 1.72, N = 6SE +/- 1.80, N = 624.1529.7732.7041.8346.3047.101. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directoryzfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 image306090120150SE +/- 3.31, N = 6SE +/- 0.03, N = 3SE +/- 0.59, N = 3SE +/- 0.41, N = 3SE +/- 0.67, N = 336.4370.7771.3774.27138.00139.001. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl

IOzone

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 8GB - Disk Test: Read Performancezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagepve-ubuntu-local-150100150200250SE +/- 1.26, N = 3SE +/- 0.58, N = 3SE +/- 1.22, N = 3SE +/- 1.25, N = 3SE +/- 0.79, N = 3SE +/- 0.57, N = 3SE +/- 10.35, N = 680.5086.07165.90167.07215.34216.63245.631. (CC) gcc options: -O3

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 8GB - Disk Test: Read Performancezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagepve-ubuntu-local-160120180240300SE +/- 2.19, N = 6SE +/- 1.27, N = 3SE +/- 0.12, N = 3SE +/- 0.79, N = 3SE +/- 0.88, N = 3SE +/- 2.04, N = 3SE +/- 13.31, N = 684.9594.99173.68174.63223.31224.74256.871. (CC) gcc options: -O3

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 8GB - Disk Test: Write Performancezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagepve-ubuntu-local-150100150200250SE +/- 1.42, N = 6SE +/- 2.01, N = 6SE +/- 0.50, N = 3SE +/- 2.02, N = 6SE +/- 0.82, N = 3SE +/- 0.85, N = 3SE +/- 1.30, N = 360.0065.2266.7972.07156.30157.79206.191. (CC) gcc options: -O3

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 8GB - Disk Test: Write Performancezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagepve-ubuntu-local-150100150200250SE +/- 1.60, N = 6SE +/- 1.02, N = 6SE +/- 1.08, N = 3SE +/- 1.13, N = 3SE +/- 1.79, N = 3SE +/- 1.03, N = 3SE +/- 2.78, N = 357.0061.4363.1074.06161.59162.04207.631. (CC) gcc options: -O3

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 8GB - Disk Test: Read Performancezfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagepve-ubuntu-local-160120180240300SE +/- 2.08, N = 6SE +/- 1.37, N = 4SE +/- 0.49, N = 3SE +/- 0.49, N = 3SE +/- 0.72, N = 3SE +/- 0.36, N = 3SE +/- 10.90, N = 684.8688.90172.38174.56225.08226.37262.941. (CC) gcc options: -O3

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 8GB - Disk Test: Write Performancezfs ssd mirror, ashift=13, recordsize 8k, raw imagezfs ssd mirror, ashift=13, recordsize 8k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, zvolzfs ssd mirror, ashift=13, recordsize 8k, zvolzfs ssd mirror, ashift=13, recordsize 64k, qcow2 imagezfs ssd mirror, ashift=13, recordsize 64k, raw imagepve-ubuntu-local-150100150200250SE +/- 0.37, N = 3SE +/- 0.92, N = 6SE +/- 1.77, N = 6SE +/- 3.14, N = 6SE +/- 2.73, N = 6SE +/- 2.94, N = 6SE +/- 3.16, N = 355.7258.8760.0367.62156.04160.28207.411. (CC) gcc options: -O3