Compare your own system(s) to this result file with the
Phoronix Test Suite by running the command:
phoronix-test-suite benchmark 1803243-KH-1803231TY84 ZFS SSD Benchmark: RAW IMAGE vs QCOW2 vs ZVOL for KVM - Phoronix Test Suite ZFS SSD Benchmark: RAW IMAGE vs QCOW2 vs ZVOL for KVM This benchmark show’s the performance of a zfs pool
providing storage to a kvm virtual machine with three different formats:
raw image files on plain dataset
qcow2 image file on plain dataset
zvol
For each of those several filesystem benchmarks from the phoronix-benchmark-suite are run with two different recordsizes:
8k
64k
Host Details:
cpu: Intel® Xeon® CPU E5320 @ 1.86GHz
ram: 6GB DDR2
sata: Intel Corporation 631xESB/632xESB SATA AHCI Controller (rev 09
os: Debian GNU/Linux 9.4 (stretch), fully upgraded as of 2018-03-15
zfs: 0.7.6-1~bpo9+1 (from stretch-backports)
ssd: 2x samsung 840 pro 256GB
KVM machine:
cpu: host
ram: 1024MB
disk: bus=scsi,cache=writeback,io=threads
os: Debian GNU/Linux 9.1 (stretch)
fs: ext4
The zfs pool used consists of a single mirrored vdev with samsung 840 pro ssd’s.
Each benchmark is run like this:
drop all caches
trim whole ssd’s
create zfs pool (ashift=13,compression=lz4,atime=on,relatime=on
benchmark,devices=off,exec=off,setuid=off,xattr=off)
create disk:
qemu-img create -f raw debian9.raw 50G
qemu-img create -f qcow2 -o cluster_size=8k,preallocation=metadata,compat=1.1,lazy_refcounts=on debian9.qcow2 50G
zfs create -o volblocksize=8k -V 50G benchmark/kvm/debian9
create kvm machine
take timestamp
let debian9 install automatically
save install time
install phoronix-test-suite and needed dependencies in the vm
run the pts
trim the filesystem and save the time it took
copy the results to the kvm host
destroy vm and zpool
other test results:
8k_raw - install time: 619s, trim: 39.401s
64k raw - install time: 589s, trim: 0m9.593s
8k qcow2 - install time: 615s, trim: 1m45.421s
64k qcow2 - install time: 590s, trim: 0m7.598s
8k zvol - install time: 621s, trim: 0m11.353s
64k zvol - install time: 605s
HTML result view exported from: https://openbenchmarking.org/result/1803243-KH-1803231TY84&sor&grw .
ZFS SSD Benchmark: RAW IMAGE vs QCOW2 vs ZVOL for KVM Processor Motherboard Chipset Memory Disk Graphics Network OS Kernel Compiler File-System Screen Resolution System Layer zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 8k, zvol pve-ubuntu-local-1 8 x Intel Celeron_4x0 (Conroe/Merom Class Core 2) @ 1.86GHz (8 Cores) QEMU Standard PC (i440FX + PIIX 1996) Intel 440FX- 82441FX PMC 1 x 1024 MB RAM QEMU 54GB QEMU HDD Cirrus Logic GD 5446 Realtek RTL-8100/8101L/8139 Debian 9.1 4.9.0-4-amd64 (x86_64) GCC 6.3.0 20170516 ext4 1024x768 qemu Common KVM @ 3.09GHz (2 Cores) 2048MB 34GB QEMU HDD Device 1234:1111 Red Hat Virtio device Ubuntu 16.04 4.4.0-116-generic (x86_64) GCC 5.4.0 20160609 800x600 OpenBenchmarking.org Compiler Details - zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 64k, raw image: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 64k, zvol: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 8k, raw image: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - zfs ssd mirror, ashift=13, recordsize 8k, zvol: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - pve-ubuntu-local-1: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --disable-werror --enable-checking=release --enable-clocale=gnu --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-tune=generic -v Disk Details - zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image: CFQ / data=ordered,errors=remount-ro,relatime,rw - zfs ssd mirror, ashift=13, recordsize 64k, raw image: CFQ / data=ordered,errors=remount-ro,relatime,rw - zfs ssd mirror, ashift=13, recordsize 64k, zvol: CFQ / data=ordered,errors=remount-ro,relatime,rw - zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image: CFQ / data=ordered,errors=remount-ro,relatime,rw - zfs ssd mirror, ashift=13, recordsize 8k, raw image: CFQ / data=ordered,errors=remount-ro,relatime,rw - zfs ssd mirror, ashift=13, recordsize 8k, zvol: CFQ / data=ordered,errors=remount-ro,relatime,rw - pve-ubuntu-local-1: DEADLINE / data=ordered,errors=remount-ro,relatime,rw System Details - zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image: Python 2.7.13. - zfs ssd mirror, ashift=13, recordsize 64k, raw image: Python 2.7.13. - zfs ssd mirror, ashift=13, recordsize 64k, zvol: Python 2.7.13. - zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image: Python 2.7.13. - zfs ssd mirror, ashift=13, recordsize 8k, raw image: Python 2.7.13. - zfs ssd mirror, ashift=13, recordsize 8k, zvol: Python 2.7.13. - pve-ubuntu-local-1: Disk Scheduler: DEADLINE. Python 2.7.12.
ZFS SSD Benchmark: RAW IMAGE vs QCOW2 vs ZVOL for KVM compilebench: Compile compilebench: Initial Create compilebench: Read Compiled Tree dbench: 1 dbench: 6 dbench: 12 aio-stress: Rand Write dbench: 48 dbench: 128 dbench: 256 iozone: 1MB - 8GB - Read Performance iozone: 4Kb - 8GB - Read Performance iozone: 1MB - 8GB - Write Performance iozone: 4Kb - 8GB - Write Performance iozone: 64Kb - 8GB - Read Performance iozone: 64Kb - 8GB - Write Performance zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 8k, zvol pve-ubuntu-local-1 142.88 82.40 64.69 42.33 102.98 162.59 125.74 294.03 203.76 118.70 215.34 223.31 156.30 161.59 225.08 156.04 147.18 86.85 64.83 42.60 105.25 166.12 131.81 292.22 207.32 120.18 216.63 224.74 157.79 162.04 226.37 160.28 81.59 50.11 60.72 42.72 88.54 133.55 111.33 177.89 102.07 62.00 86.07 94.99 66.79 63.10 88.90 60.03 61.65 42.54 56.72 42.20 89.99 137.50 93.70 176.41 89.75 55.57 165.90 173.68 65.22 61.43 174.56 58.87 72.13 48.40 55.72 42.18 91.96 137.66 98.01 181.38 97.79 55.77 167.07 174.63 60.00 57.00 172.38 55.72 91.69 49.32 46.58 40.87 86.79 128.76 100.87 153.63 88.41 58.47 80.50 84.95 72.07 74.06 84.86 67.62 262.57 182.82 166.58 430.74 826.51 779.05 49.15 792.05 308.19 201.67 245.63 256.87 206.19 207.63 262.94 207.41 OpenBenchmarking.org
Compile Bench Test: Compile OpenBenchmarking.org MB/s, More Is Better Compile Bench 0.6 Test: Compile pve-ubuntu-local-1 zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, zvol zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image 60 120 180 240 300 SE +/- 4.39, N = 6 SE +/- 5.21, N = 6 SE +/- 5.01, N = 6 SE +/- 2.73, N = 6 SE +/- 1.40, N = 3 SE +/- 2.12, N = 6 SE +/- 4.97, N = 6 262.57 147.18 142.88 91.69 81.59 72.13 61.65
Compile Bench Test: Initial Create OpenBenchmarking.org MB/s, More Is Better Compile Bench 0.6 Test: Initial Create pve-ubuntu-local-1 zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, zvol zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image 40 80 120 160 200 SE +/- 10.19, N = 3 SE +/- 3.08, N = 3 SE +/- 6.42, N = 3 SE +/- 5.96, N = 3 SE +/- 3.57, N = 3 SE +/- 1.91, N = 3 SE +/- 8.32, N = 3 182.82 86.85 82.40 50.11 49.32 48.40 42.54
Compile Bench Test: Read Compiled Tree OpenBenchmarking.org MB/s, More Is Better Compile Bench 0.6 Test: Read Compiled Tree pve-ubuntu-local-1 zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 8k, zvol 40 80 120 160 200 SE +/- 2.00, N = 3 SE +/- 0.33, N = 3 SE +/- 0.14, N = 3 SE +/- 2.51, N = 3 SE +/- 0.52, N = 3 SE +/- 0.15, N = 3 SE +/- 0.65, N = 3 166.58 64.83 64.69 60.72 56.72 55.72 46.58
Dbench Client Count: 1 OpenBenchmarking.org MB/s, More Is Better Dbench 4.0 Client Count: 1 pve-ubuntu-local-1 zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 8k, zvol 90 180 270 360 450 SE +/- 3.32, N = 3 SE +/- 0.01, N = 3 SE +/- 0.09, N = 3 SE +/- 0.08, N = 3 SE +/- 0.02, N = 3 SE +/- 0.05, N = 3 SE +/- 0.15, N = 3 430.74 42.72 42.60 42.33 42.20 42.18 40.87 1. (CC) gcc options: -lpopt -O2
Dbench Client Count: 6 OpenBenchmarking.org MB/s, More Is Better Dbench 4.0 Client Count: 6 pve-ubuntu-local-1 zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, zvol 200 400 600 800 1000 SE +/- 15.67, N = 3 SE +/- 1.07, N = 3 SE +/- 0.91, N = 3 SE +/- 1.36, N = 3 SE +/- 0.90, N = 3 SE +/- 1.27, N = 3 SE +/- 0.16, N = 3 826.51 105.25 102.98 91.96 89.99 88.54 86.79 1. (CC) gcc options: -lpopt -O2
Dbench Client Count: 12 OpenBenchmarking.org MB/s, More Is Better Dbench 4.0 Client Count: 12 pve-ubuntu-local-1 zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, zvol 200 400 600 800 1000 SE +/- 9.36, N = 3 SE +/- 0.58, N = 3 SE +/- 0.75, N = 3 SE +/- 1.52, N = 3 SE +/- 1.47, N = 3 SE +/- 0.41, N = 3 SE +/- 0.84, N = 3 779.05 166.12 162.59 137.66 137.50 133.55 128.76 1. (CC) gcc options: -lpopt -O2
AIO-Stress Test: Random Write OpenBenchmarking.org MB/s, More Is Better AIO-Stress 0.21 Test: Random Write zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, zvol zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image pve-ubuntu-local-1 30 60 90 120 150 SE +/- 0.94, N = 3 SE +/- 0.78, N = 3 SE +/- 0.51, N = 3 SE +/- 3.01, N = 6 SE +/- 0.50, N = 3 SE +/- 1.19, N = 3 SE +/- 0.83, N = 4 131.81 125.74 111.33 100.87 98.01 93.70 49.15 1. (CC) gcc options: -pthread -laio
Dbench Client Count: 48 OpenBenchmarking.org MB/s, More Is Better Dbench 4.0 Client Count: 48 pve-ubuntu-local-1 zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, zvol 200 400 600 800 1000 SE +/- 2.73, N = 3 SE +/- 4.38, N = 3 SE +/- 4.21, N = 3 SE +/- 7.08, N = 6 SE +/- 6.43, N = 6 SE +/- 7.08, N = 6 SE +/- 6.84, N = 6 792.05 294.03 292.22 181.38 177.89 176.41 153.63 1. (CC) gcc options: -lpopt -O2
Dbench Client Count: 128 OpenBenchmarking.org MB/s, More Is Better Dbench 4.0 Client Count: 128 pve-ubuntu-local-1 zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, zvol 70 140 210 280 350 SE +/- 6.16, N = 3 SE +/- 2.00, N = 3 SE +/- 0.99, N = 3 SE +/- 4.96, N = 6 SE +/- 5.85, N = 6 SE +/- 5.76, N = 6 SE +/- 4.52, N = 6 308.19 207.32 203.76 102.07 97.79 89.75 88.41 1. (CC) gcc options: -lpopt -O2
Dbench Client Count: 256 OpenBenchmarking.org MB/s, More Is Better Dbench 4.0 Client Count: 256 pve-ubuntu-local-1 zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, zvol zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image 40 80 120 160 200 SE +/- 1.90, N = 3 SE +/- 0.40, N = 3 SE +/- 0.23, N = 3 SE +/- 2.00, N = 6 SE +/- 1.39, N = 6 SE +/- 2.11, N = 6 SE +/- 0.85, N = 3 201.67 120.18 118.70 62.00 58.47 55.77 55.57 1. (CC) gcc options: -lpopt -O2
IOzone Record Size: 1MB - File Size: 8GB - Disk Test: Read Performance OpenBenchmarking.org MB/s, More Is Better IOzone 3.465 Record Size: 1MB - File Size: 8GB - Disk Test: Read Performance pve-ubuntu-local-1 zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, zvol 50 100 150 200 250 SE +/- 10.35, N = 6 SE +/- 0.57, N = 3 SE +/- 0.79, N = 3 SE +/- 1.25, N = 3 SE +/- 1.22, N = 3 SE +/- 0.58, N = 3 SE +/- 1.26, N = 3 245.63 216.63 215.34 167.07 165.90 86.07 80.50 1. (CC) gcc options: -O3
IOzone Record Size: 4Kb - File Size: 8GB - Disk Test: Read Performance OpenBenchmarking.org MB/s, More Is Better IOzone 3.465 Record Size: 4Kb - File Size: 8GB - Disk Test: Read Performance pve-ubuntu-local-1 zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, zvol 60 120 180 240 300 SE +/- 13.31, N = 6 SE +/- 2.04, N = 3 SE +/- 0.88, N = 3 SE +/- 0.79, N = 3 SE +/- 0.12, N = 3 SE +/- 1.27, N = 3 SE +/- 2.19, N = 6 256.87 224.74 223.31 174.63 173.68 94.99 84.95 1. (CC) gcc options: -O3
IOzone Record Size: 1MB - File Size: 8GB - Disk Test: Write Performance OpenBenchmarking.org MB/s, More Is Better IOzone 3.465 Record Size: 1MB - File Size: 8GB - Disk Test: Write Performance pve-ubuntu-local-1 zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, zvol zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, raw image 50 100 150 200 250 SE +/- 1.30, N = 3 SE +/- 0.85, N = 3 SE +/- 0.82, N = 3 SE +/- 2.02, N = 6 SE +/- 0.50, N = 3 SE +/- 2.01, N = 6 SE +/- 1.42, N = 6 206.19 157.79 156.30 72.07 66.79 65.22 60.00 1. (CC) gcc options: -O3
IOzone Record Size: 4Kb - File Size: 8GB - Disk Test: Write Performance OpenBenchmarking.org MB/s, More Is Better IOzone 3.465 Record Size: 4Kb - File Size: 8GB - Disk Test: Write Performance pve-ubuntu-local-1 zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, zvol zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, raw image 50 100 150 200 250 SE +/- 2.78, N = 3 SE +/- 1.03, N = 3 SE +/- 1.79, N = 3 SE +/- 1.13, N = 3 SE +/- 1.08, N = 3 SE +/- 1.02, N = 6 SE +/- 1.60, N = 6 207.63 162.04 161.59 74.06 63.10 61.43 57.00 1. (CC) gcc options: -O3
IOzone Record Size: 64Kb - File Size: 8GB - Disk Test: Read Performance OpenBenchmarking.org MB/s, More Is Better IOzone 3.465 Record Size: 64Kb - File Size: 8GB - Disk Test: Read Performance pve-ubuntu-local-1 zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, zvol 60 120 180 240 300 SE +/- 10.90, N = 6 SE +/- 0.36, N = 3 SE +/- 0.72, N = 3 SE +/- 0.49, N = 3 SE +/- 0.49, N = 3 SE +/- 1.37, N = 4 SE +/- 2.08, N = 6 262.94 226.37 225.08 174.56 172.38 88.90 84.86 1. (CC) gcc options: -O3
IOzone Record Size: 64Kb - File Size: 8GB - Disk Test: Write Performance OpenBenchmarking.org MB/s, More Is Better IOzone 3.465 Record Size: 64Kb - File Size: 8GB - Disk Test: Write Performance pve-ubuntu-local-1 zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, zvol zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, raw image 50 100 150 200 250 SE +/- 3.16, N = 3 SE +/- 2.94, N = 6 SE +/- 2.73, N = 6 SE +/- 3.14, N = 6 SE +/- 1.77, N = 6 SE +/- 0.92, N = 6 SE +/- 0.37, N = 3 207.41 160.28 156.04 67.62 60.03 58.87 55.72 1. (CC) gcc options: -O3
Flexible IO Tester Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.1 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory zfs ssd mirror, ashift=13, recordsize 8k, zvol zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image 100 200 300 400 500 SE +/- 7.46, N = 6 SE +/- 10.34, N = 6 SE +/- 2.08, N = 3 SE +/- 1.20, N = 3 SE +/- 1.00, N = 3 SE +/- 1.00, N = 3 473 454 294 292 230 228 1. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl
Flexible IO Tester Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.1 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, zvol zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, raw image 9 18 27 36 45 SE +/- 0.52, N = 3 SE +/- 0.65, N = 4 SE +/- 0.38, N = 3 SE +/- 0.03, N = 3 SE +/- 0.10, N = 3 SE +/- 0.40, N = 3 40.73 40.65 39.20 38.93 38.60 36.80 1. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl
Flexible IO Tester Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.1 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, zvol zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, raw image 50 100 150 200 250 SE +/- 1.45, N = 3 SE +/- 2.52, N = 3 SE +/- 0.88, N = 3 SE +/- 0.58, N = 3 239 237 232 230 230 229 1. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl
Flexible IO Tester Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.1 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, zvol 30 60 90 120 150 SE +/- 0.58, N = 3 SE +/- 0.67, N = 3 SE +/- 0.31, N = 3 SE +/- 0.09, N = 3 SE +/- 2.09, N = 6 SE +/- 1.08, N = 6 146.00 143.00 71.20 70.23 54.75 42.05 1. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl
Flexible IO Tester Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.1 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, zvol zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 64k, zvol 7 14 21 28 35 SE +/- 0.20, N = 3 SE +/- 0.19, N = 3 SE +/- 0.10, N = 3 SE +/- 0.26, N = 3 SE +/- 0.24, N = 3 SE +/- 0.21, N = 4 29.20 25.73 20.70 17.20 15.97 12.05 1. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl
Flexible IO Tester Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.1 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, zvol 30 60 90 120 150 SE +/- 0.88, N = 3 SE +/- 0.67, N = 3 SE +/- 0.81, N = 3 SE +/- 0.03, N = 3 SE +/- 0.25, N = 3 SE +/- 3.30, N = 6 118.00 113.00 66.40 61.33 59.40 38.75 1. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl
Flexible IO Tester Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.1 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory zfs ssd mirror, ashift=13, recordsize 8k, zvol zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image 100 200 300 400 500 SE +/- 10.48, N = 6 SE +/- 5.61, N = 3 SE +/- 1.20, N = 3 SE +/- 0.67, N = 3 SE +/- 0.67, N = 3 472 458 291 289 229 226 1. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl
Flexible IO Tester Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.1 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, zvol zfs ssd mirror, ashift=13, recordsize 64k, zvol 13 26 39 52 65 SE +/- 2.11, N = 6 SE +/- 0.23, N = 3 SE +/- 1.33, N = 6 SE +/- 1.52, N = 6 SE +/- 1.10, N = 6 SE +/- 0.32, N = 3 58.32 55.50 54.17 45.85 45.02 44.73 1. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl
Flexible IO Tester Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.1 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory zfs ssd mirror, ashift=13, recordsize 8k, zvol zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image 80 160 240 320 400 SE +/- 4.51, N = 3 SE +/- 7.51, N = 6 SE +/- 1.33, N = 3 SE +/- 1.76, N = 3 SE +/- 1.33, N = 3 SE +/- 0.58, N = 3 353 347 306 303 252 244 1. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl
Flexible IO Tester Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.1 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, zvol 30 60 90 120 150 SE +/- 2.33, N = 3 SE +/- 0.93, N = 3 SE +/- 1.08, N = 3 SE +/- 1.94, N = 6 SE +/- 6.42, N = 6 152.00 147.00 74.37 72.03 66.28 45.33 1. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl
Flexible IO Tester Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.1 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, zvol 11 22 33 44 55 SE +/- 1.80, N = 6 SE +/- 1.72, N = 6 SE +/- 1.78, N = 6 SE +/- 1.80, N = 6 SE +/- 0.64, N = 6 SE +/- 0.57, N = 6 47.10 46.30 41.83 32.70 29.77 24.15 1. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl
Flexible IO Tester Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.1 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image zfs ssd mirror, ashift=13, recordsize 64k, raw image zfs ssd mirror, ashift=13, recordsize 8k, raw image zfs ssd mirror, ashift=13, recordsize 64k, zvol zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image zfs ssd mirror, ashift=13, recordsize 8k, zvol 30 60 90 120 150 SE +/- 0.67, N = 3 SE +/- 0.41, N = 3 SE +/- 0.59, N = 3 SE +/- 0.03, N = 3 SE +/- 3.31, N = 6 139.00 138.00 74.27 71.37 70.77 36.43 1. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -lrt -laio -lm -lpthread -ldl
Phoronix Test Suite v10.8.4