ZFS SSD Benchmark: RAW IMAGE vs QCOW2 vs ZVOL for KVM FNAS zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image: Processor: 8 x Intel Celeron_4x0 (Conroe/Merom Class Core 2) @ 1.86GHz (8 Cores), Motherboard: QEMU Standard PC (i440FX + PIIX 1996), Chipset: Intel 440FX- 82441FX PMC, Memory: 1 x 1024 MB RAM QEMU, Disk: 54GB QEMU HDD, Graphics: Cirrus Logic GD 5446, Network: Realtek RTL-8100/8101L/8139 OS: Debian 9.1, Kernel: 4.9.0-4-amd64 (x86_64), Compiler: GCC 6.3.0 20170516, File-System: ext4, Screen Resolution: 1024x768, System Layer: qemu zfs ssd mirror, ashift=13, recordsize 64k, raw image: Processor: 8 x Intel Celeron_4x0 (Conroe/Merom Class Core 2) @ 1.86GHz (8 Cores), Motherboard: QEMU Standard PC (i440FX + PIIX 1996), Chipset: Intel 440FX- 82441FX PMC, Memory: 1 x 1024 MB RAM QEMU, Disk: 54GB QEMU HDD, Graphics: Cirrus Logic GD 5446, Network: Realtek RTL-8100/8101L/8139 OS: Debian 9.1, Kernel: 4.9.0-4-amd64 (x86_64), Compiler: GCC 6.3.0 20170516, File-System: ext4, Screen Resolution: 1024x768, System Layer: qemu zfs ssd mirror, ashift=13, recordsize 64k, zvol: Processor: 8 x Intel Celeron_4x0 (Conroe/Merom Class Core 2) @ 1.86GHz (8 Cores), Motherboard: QEMU Standard PC (i440FX + PIIX 1996), Chipset: Intel 440FX- 82441FX PMC, Memory: 1 x 1024 MB RAM QEMU, Disk: 54GB QEMU HDD, Graphics: Cirrus Logic GD 5446, Network: Realtek RTL-8100/8101L/8139 OS: Debian 9.1, Kernel: 4.9.0-4-amd64 (x86_64), Compiler: GCC 6.3.0 20170516, File-System: ext4, Screen Resolution: 1024x768, System Layer: qemu zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image: Processor: 8 x Intel Celeron_4x0 (Conroe/Merom Class Core 2) @ 1.86GHz (8 Cores), Motherboard: QEMU Standard PC (i440FX + PIIX 1996), Chipset: Intel 440FX- 82441FX PMC, Memory: 1 x 1024 MB RAM QEMU, Disk: 54GB QEMU HDD, Graphics: Cirrus Logic GD 5446, Network: Realtek RTL-8100/8101L/8139 OS: Debian 9.1, Kernel: 4.9.0-4-amd64 (x86_64), Compiler: GCC 6.3.0 20170516, File-System: ext4, Screen Resolution: 1024x768, System Layer: qemu zfs ssd mirror, ashift=13, recordsize 8k, raw image: Processor: 8 x Intel Celeron_4x0 (Conroe/Merom Class Core 2) @ 1.86GHz (8 Cores), Motherboard: QEMU Standard PC (i440FX + PIIX 1996), Chipset: Intel 440FX- 82441FX PMC, Memory: 1 x 1024 MB RAM QEMU, Disk: 54GB QEMU HDD, Graphics: Cirrus Logic GD 5446, Network: Realtek RTL-8100/8101L/8139 OS: Debian 9.1, Kernel: 4.9.0-4-amd64 (x86_64), Compiler: GCC 6.3.0 20170516, File-System: ext4, Screen Resolution: 1024x768, System Layer: qemu zfs ssd mirror, ashift=13, recordsize 8k, zvol: Processor: 8 x Intel Celeron_4x0 (Conroe/Merom Class Core 2) @ 1.86GHz (8 Cores), Motherboard: QEMU Standard PC (i440FX + PIIX 1996), Chipset: Intel 440FX- 82441FX PMC, Memory: 1 x 1024 MB RAM QEMU, Disk: 54GB QEMU HDD, Graphics: Cirrus Logic GD 5446, Network: Realtek RTL-8100/8101L/8139 OS: Debian 9.1, Kernel: 4.9.0-4-amd64 (x86_64), Compiler: GCC 6.3.0 20170516, File-System: ext4, Screen Resolution: 1024x768, System Layer: qemu pve-ubuntu-fnas-1: Processor: Common KVM @ 3.09GHz (2 Cores), Motherboard: QEMU Standard PC (i440FX + PIIX 1996), Chipset: Intel 440FX- 82441FX PMC, Memory: 2048MB, Disk: 537GB QEMU HDD, Graphics: Device 1234:1111, Network: Red Hat Virtio device OS: Ubuntu 16.04, Kernel: 4.4.0-116-generic (x86_64), Compiler: GCC 5.4.0 20160609, File-System: ext4, Screen Resolution: 800x600, System Layer: qemu AIO-Stress 0.21 Test: Random Write MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 125.74 |= zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 131.81 |= zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 111.33 |= zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 93.70 |= zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 98.01 |= zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 100.87 |= pve-ubuntu-fnas-1 ...................................... 1449.77 |============= Flexible IO Tester 3.1 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 292 |========== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 294 |=========== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 454 |================ zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 228 |======== zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 230 |======== zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 473 |================= Flexible IO Tester 3.1 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 38.60 |============== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 38.93 |============== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 40.73 |=============== zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 40.65 |=============== zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 36.80 |============== zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 39.20 |============== Flexible IO Tester 3.1 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 237 |================= zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 239 |================= zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 230 |================ zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 232 |================= zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 229 |================ zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 230 |================ Flexible IO Tester 3.1 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 143.00 |============== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 146.00 |============== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 54.75 |===== zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 70.23 |======= zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 71.20 |======= zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 42.05 |==== Flexible IO Tester 3.1 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 15.97 |======== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 17.20 |========= zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 12.05 |====== zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 25.73 |============= zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 29.20 |=============== zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 20.70 |=========== Flexible IO Tester 3.1 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 113.00 |============= zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 118.00 |============== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 66.40 |======== zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 59.40 |======= zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 61.33 |======= zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 38.75 |===== Flexible IO Tester 3.1 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 291 |========== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 289 |========== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 458 |================ zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 226 |======== zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 229 |======== zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 472 |================= Flexible IO Tester 3.1 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 54.17 |============== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 55.50 |============== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 44.73 |============ zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 45.85 |============ zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 58.32 |=============== zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 45.02 |============ Flexible IO Tester 3.1 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 306 |=============== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 303 |=============== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 347 |================= zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 244 |============ zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 252 |============ zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 353 |================= Flexible IO Tester 3.1 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 147.00 |============== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 152.00 |============== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 66.28 |====== zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 72.03 |======= zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 74.37 |======= zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 45.33 |==== Flexible IO Tester 3.1 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 46.30 |=============== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 47.10 |=============== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 29.77 |========= zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 32.70 |========== zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 41.83 |============= zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 24.15 |======== Flexible IO Tester 3.1 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 139.00 |============== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 138.00 |============== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 71.37 |======= zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 70.77 |======= zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 74.27 |======= zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 36.43 |==== Dbench 4.0 Client Count: 1 MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 42.33 |=== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 42.60 |=== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 42.72 |=== zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 42.20 |=== zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 42.18 |=== zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 40.87 |=== pve-ubuntu-fnas-1 ...................................... 179.20 |============== Dbench 4.0 Client Count: 6 MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 102.98 |==== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 105.25 |===== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 88.54 |==== zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 89.99 |==== zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 91.96 |==== zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 86.79 |==== pve-ubuntu-fnas-1 ...................................... 320.86 |============== Dbench 4.0 Client Count: 12 MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 162.59 |====== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 166.12 |====== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 133.55 |===== zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 137.50 |===== zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 137.66 |===== zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 128.76 |===== pve-ubuntu-fnas-1 ...................................... 387.51 |============== Dbench 4.0 Client Count: 48 MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 294.03 |========= zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 292.22 |========= zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 177.89 |====== zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 176.41 |====== zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 181.38 |====== zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 153.63 |===== pve-ubuntu-fnas-1 ...................................... 437.60 |============== Dbench 4.0 Client Count: 128 MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 203.76 |=========== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 207.32 |=========== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 102.07 |===== zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 89.75 |===== zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 97.79 |===== zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 88.41 |===== pve-ubuntu-fnas-1 ...................................... 259.95 |============== Dbench 4.0 Client Count: 256 MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 118.70 |===== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 120.18 |===== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 62.00 |=== zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 55.57 |== zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 55.77 |== zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 58.47 |== pve-ubuntu-fnas-1 ...................................... 333.19 |============== IOzone 3.465 Record Size: 1MB - File Size: 8GB - Disk Test: Read Performance MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 215.34 |============== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 216.63 |============== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 86.07 |====== zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 165.90 |=========== zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 167.07 |=========== zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 80.50 |===== pve-ubuntu-fnas-1 ...................................... 111.88 |======= IOzone 3.465 Record Size: 4Kb - File Size: 8GB - Disk Test: Read Performance MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 223.31 |============== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 224.74 |============== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 94.99 |====== zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 173.68 |=========== zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 174.63 |=========== zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 84.95 |===== pve-ubuntu-fnas-1 ...................................... 112.02 |======= IOzone 3.465 Record Size: 1MB - File Size: 8GB - Disk Test: Write Performance MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 156.30 |============== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 157.79 |============== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 66.79 |====== zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 65.22 |====== zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 60.00 |===== zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 72.07 |====== pve-ubuntu-fnas-1 ...................................... 111.94 |========== IOzone 3.465 Record Size: 4Kb - File Size: 8GB - Disk Test: Write Performance MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 161.59 |============== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 162.04 |============== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 63.10 |===== zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 61.43 |===== zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 57.00 |===== zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 74.06 |====== pve-ubuntu-fnas-1 ...................................... 111.83 |========== IOzone 3.465 Record Size: 64Kb - File Size: 8GB - Disk Test: Read Performance MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 225.08 |============== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 226.37 |============== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 88.90 |===== zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 174.56 |=========== zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 172.38 |=========== zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 84.86 |===== pve-ubuntu-fnas-1 ...................................... 112.04 |======= IOzone 3.465 Record Size: 64Kb - File Size: 8GB - Disk Test: Write Performance MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 156.04 |============== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 160.28 |============== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 60.03 |===== zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 58.87 |===== zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 55.72 |===== zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 67.62 |====== pve-ubuntu-fnas-1 ...................................... 111.94 |========== Compile Bench 0.6 Test: Compile MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 142.88 |============== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 147.18 |============== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 81.59 |======== zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 61.65 |====== zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 72.13 |======= zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 91.69 |========= pve-ubuntu-fnas-1 ...................................... 117.05 |=========== Compile Bench 0.6 Test: Initial Create MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 82.40 |=========== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 86.85 |============ zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 50.11 |======= zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 42.54 |====== zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 48.40 |======= zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 49.32 |======= pve-ubuntu-fnas-1 ...................................... 100.89 |============== Compile Bench 0.6 Test: Read Compiled Tree MB/s > Higher Is Better zfs ssd mirror, ashift=13, recordsize 64k, qcow2 image . 64.69 |=============== zfs ssd mirror, ashift=13, recordsize 64k, raw image ... 64.83 |=============== zfs ssd mirror, ashift=13, recordsize 64k, zvol ........ 60.72 |============== zfs ssd mirror, ashift=13, recordsize 8k, qcow2 image .. 56.72 |============= zfs ssd mirror, ashift=13, recordsize 8k, raw image .... 55.72 |============= zfs ssd mirror, ashift=13, recordsize 8k, zvol ......... 46.58 |=========== pve-ubuntu-fnas-1 ...................................... 64.91 |===============