ZFS + KVM + NTFS Benchmarking Various Record Sizes This benchmark aims to identify optimal combination of record sizes (i.e., recordsize for ZFS, cluster_size for QCOW2, allocation unit size for NTFS) for a Windows 10 Guest VM running on top of ZFS + KVM. Host: Dual Intel Xeon 6130 @ 2.10GHz, 384GB RAM, CentOS 7.6 Virtualization Base Environment, ZFS 0.8.0 via EPEL zfs-testing repository ZFS Config: 4 x RAIDZ2 (5 x 8TB 7.2K HDD) + 2 x OPTANE 900 280GB SLOG MIRROR + OPTANE 900 480GB CACHE Windows Guest: 2 vCPUs, 4GB RAM, 40 GB QCOW2 (bus:virtio, cache:writeback, io:default, cluster size 4k ~ 128k) ZFS 128k - QCOW2 64k - NTFS 4k: Processor: 2 x Intel Xeon (Skylake IBRS) @ 2.10GHz (2 Cores), Motherboard: (0.5.1 BIOS), Memory: 1 x 4096 MB, Disk: 40GB Red Hat VirtIO, Graphics: Microsoft Basic Display OS: Microsoft Windows 10 Enterprise Evaluation Build 18362, Kernel: 10.0 (x86_64), Display Driver: 10.0.18362.1, File-System: NTFS, Screen Resolution: 1024x768 ZFS 64k - QCOW2 64k - NTFS 64k: Processor: 2 x Intel Xeon (Skylake IBRS) @ 2.10GHz (2 Cores), Motherboard: (0.5.1 BIOS), Memory: 1 x 4096 MB, Disk: 40GB Red Hat VirtIO, Graphics: Microsoft Basic Display OS: Microsoft Windows 10 Enterprise Evaluation Build 18362, Kernel: 10.0 (x86_64), Display Driver: 10.0.18362.1, File-System: NTFS, Screen Resolution: 1024x768 ZFS 16k - QCOW2 16k - NTFS 16k: Processor: 2 x Intel Xeon (Skylake IBRS) @ 2.10GHz (2 Cores), Motherboard: (0.5.1 BIOS), Memory: 1 x 4096 MB, Disk: 40GB Red Hat VirtIO, Graphics: Microsoft Basic Display OS: Microsoft Windows 10 Enterprise Evaluation Build 18362, Kernel: 10.0 (x86_64), Display Driver: 10.0.18362.1, File-System: NTFS, Screen Resolution: 1024x768 ZFS 4k - QCOW2 4k - NTFS - 4k: Processor: 2 x Intel Xeon (Skylake IBRS) @ 2.10GHz (2 Cores), Motherboard: (0.5.1 BIOS), Memory: 1 x 4096 MB, Disk: 40GB Red Hat VirtIO, Graphics: Microsoft Basic Display OS: Microsoft Windows 10 Enterprise Evaluation Build 18362, Kernel: 10.0 (x86_64), Display Driver: 10.0.18362.1, File-System: NTFS, Screen Resolution: 1024x768 ZFS 128k - QCOW2 128k - NTFS 64k: Processor: 2 x Intel Xeon (Skylake IBRS) @ 2.10GHz (2 Cores), Motherboard: (0.5.1 BIOS), Memory: 1 x 4096 MB, Disk: 40GB Red Hat VirtIO, Graphics: Microsoft Basic Display OS: Microsoft Windows 10 Enterprise Evaluation Build 18362, Kernel: 10.0 (x86_64), Display Driver: 10.0.18362.1, File-System: NTFS, Screen Resolution: 1024x768 ZFS 128k - QCOW2 128k - NTFS 4k: Processor: 2 x Intel Xeon (Skylake IBRS) @ 2.10GHz (2 Cores), Motherboard: (0.5.1 BIOS), Memory: 1 x 4096 MB, Disk: 40GB Red Hat VirtIO, Graphics: Microsoft Basic Display OS: Microsoft Windows 10 Enterprise Evaluation Build 18362, Kernel: 10.0 (x86_64), Display Driver: 10.0.18362.1, File-System: NTFS, Screen Resolution: 1024x768 ZFS 128k - QCOW2 64k - NTFS 64k: Processor: 2 x Intel Xeon (Skylake IBRS) @ 2.10GHz (2 Cores), Motherboard: (0.5.1 BIOS), Memory: 1 x 4096 MB, Disk: 40GB Red Hat VirtIO, Graphics: Microsoft Basic Display OS: Microsoft Windows 10 Enterprise Evaluation Build 18362, Kernel: 10.0 (x86_64), Display Driver: 10.0.18362.1, File-System: NTFS, Screen Resolution: 1024x768 ZFS 64k - QCOW2 64K - NTFS 4k: Processor: 2 x Intel Xeon (Skylake IBRS) @ 2.10GHz (2 Cores), Motherboard: (0.5.1 BIOS), Memory: 1 x 4096 MB, Disk: 40GB Red Hat VirtIO, Graphics: Microsoft Basic Display OS: Microsoft Windows 10 Enterprise Evaluation Build 18362, Kernel: 10.0 (x86_64), Display Driver: 10.0.18362.1, File-System: NTFS, Screen Resolution: 1024x768 ZFS 16k - QCOW2 16k - NTFS 4k: Processor: 2 x Intel Xeon (Skylake IBRS) @ 2.10GHz (2 Cores), Motherboard: (0.5.1 BIOS), Memory: 1 x 4096 MB, Disk: 40GB Red Hat VirtIO, Graphics: Microsoft Basic Display OS: Microsoft Windows 10 Enterprise Evaluation Build 18362, Kernel: 10.0 (x86_64), Display Driver: 10.0.18362.1, File-System: NTFS, Screen Resolution: 1024x768 ZFS 64k - QCOW2 64k - NTFS 16k: Processor: 2 x Intel Xeon (Skylake IBRS) @ 2.10GHz (2 Cores), Motherboard: (0.5.1 BIOS), Memory: 1 x 4096 MB, Disk: 40GB Red Hat VirtIO, Graphics: Microsoft Basic Display OS: Microsoft Windows 10 Enterprise Evaluation Build 18362, Kernel: 10.0 (x86_64), Display Driver: 10.0.18362.1, File-System: NTFS, Screen Resolution: 1024x768 ZFS 64k - QCOW2 16k - NTFS 16k: Processor: 2 x Intel Xeon (Skylake IBRS) @ 2.10GHz (2 Cores), Motherboard: (0.5.1 BIOS), Memory: 1 x 4096 MB, Disk: 40GB Red Hat VirtIO, Graphics: Microsoft Basic Display OS: Microsoft Windows 10 Enterprise Evaluation Build 18362, Kernel: 10.0 (x86_64), Display Driver: 10.0.18362.1, File-System: NTFS, Screen Resolution: 1024x768 ZFS 64k - QCOW2 4k - NTFS 4k: Processor: 2 x Intel Xeon (Skylake IBRS) @ 2.10GHz (2 Cores), Motherboard: (0.5.1 BIOS), Memory: 1 x 4096 MB, Disk: 40GB Red Hat VirtIO, Graphics: Microsoft Basic Display OS: Microsoft Windows 10 Enterprise Evaluation Build 18362, Kernel: 10.0 (x86_64), Display Driver: 10.0.18362.1, File-System: NTFS, Screen Resolution: 1024x768 ZFS 64k - QCOW2 16k - NTFS 4k: Processor: 2 x Intel Xeon (Skylake IBRS) @ 2.10GHz (2 Cores), Motherboard: (0.5.1 BIOS), Memory: 1 x 4096 MB, Disk: 40GB Red Hat VirtIO, Graphics: Microsoft Basic Display OS: Microsoft Windows 10 Enterprise Evaluation Build 18362, Kernel: 10.0 (x86_64), Display Driver: 10.0.18362.1, File-System: NTFS, Screen Resolution: 1024x768 Flexible IO Tester 3.1 Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 12500 |==================================== ZFS 64k - QCOW2 64k - NTFS 64k ... 11500 |================================= ZFS 16k - QCOW2 16k - NTFS 16k ... 7675 |====================== ZFS 4k - QCOW2 4k - NTFS - 4k .... 4191 |============ ZFS 128k - QCOW2 128k - NTFS 64k . 12160 |=================================== ZFS 128k - QCOW2 128k - NTFS 4k .. 12533 |==================================== ZFS 128k - QCOW2 64k - NTFS 64k .. 12554 |==================================== ZFS 64k - QCOW2 64K - NTFS 4k .... 12875 |===================================== ZFS 16k - QCOW2 16k - NTFS 4k .... 9830 |============================ ZFS 64k - QCOW2 64k - NTFS 16k ... 12853 |===================================== ZFS 64k - QCOW2 16k - NTFS 16k ... 9859 |============================ ZFS 64k - QCOW2 4k - NTFS 4k ..... 5270 |=============== ZFS 64k - QCOW2 16k - NTFS 4k .... 9007 |========================== Flexible IO Tester 3.1 Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 204.00 |=============================== ZFS 64k - QCOW2 64k - NTFS 64k ... 234.00 |==================================== ZFS 16k - QCOW2 16k - NTFS 16k ... 146.00 |====================== ZFS 4k - QCOW2 4k - NTFS - 4k .... 104.47 |================ ZFS 128k - QCOW2 128k - NTFS 64k . 205.00 |================================ ZFS 128k - QCOW2 128k - NTFS 4k .. 209.00 |================================ ZFS 128k - QCOW2 64k - NTFS 64k .. 208.00 |================================ ZFS 64k - QCOW2 64K - NTFS 4k .... 211.00 |================================ ZFS 16k - QCOW2 16k - NTFS 4k .... 126.00 |=================== ZFS 64k - QCOW2 64k - NTFS 16k ... 211.00 |================================ ZFS 64k - QCOW2 16k - NTFS 16k ... 160.00 |========================= ZFS 64k - QCOW2 4k - NTFS 4k ..... 110.00 |================= ZFS 64k - QCOW2 16k - NTFS 4k .... 153.00 |======================== Flexible IO Tester 3.1 Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 416 |=================================== ZFS 64k - QCOW2 64k - NTFS 64k ... 463 |======================================= ZFS 16k - QCOW2 16k - NTFS 16k ... 288 |======================== ZFS 4k - QCOW2 4k - NTFS - 4k .... 198 |================= ZFS 128k - QCOW2 128k - NTFS 64k . 408 |================================== ZFS 128k - QCOW2 128k - NTFS 4k .. 416 |=================================== ZFS 128k - QCOW2 64k - NTFS 64k .. 410 |=================================== ZFS 64k - QCOW2 64K - NTFS 4k .... 408 |================================== ZFS 16k - QCOW2 16k - NTFS 4k .... 252 |===================== ZFS 64k - QCOW2 64k - NTFS 16k ... 420 |=================================== ZFS 64k - QCOW2 16k - NTFS 16k ... 324 |=========================== ZFS 64k - QCOW2 4k - NTFS 4k ..... 228 |=================== ZFS 64k - QCOW2 16k - NTFS 4k .... 294 |========================= Flexible IO Tester 3.1 Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 838 |======================================= ZFS 64k - QCOW2 64k - NTFS 64k ... 831 |======================================= ZFS 16k - QCOW2 16k - NTFS 16k ... 578 |=========================== ZFS 4k - QCOW2 4k - NTFS - 4k .... 433 |==================== ZFS 128k - QCOW2 128k - NTFS 64k . 808 |====================================== ZFS 128k - QCOW2 128k - NTFS 4k .. 828 |======================================= ZFS 128k - QCOW2 64k - NTFS 64k .. 815 |====================================== ZFS 64k - QCOW2 64K - NTFS 4k .... 791 |===================================== ZFS 16k - QCOW2 16k - NTFS 4k .... 500 |======================= ZFS 64k - QCOW2 64k - NTFS 16k ... 830 |======================================= ZFS 64k - QCOW2 16k - NTFS 16k ... 640 |============================== ZFS 64k - QCOW2 4k - NTFS 4k ..... 397 |================== ZFS 64k - QCOW2 16k - NTFS 4k .... 592 |============================ Flexible IO Tester 3.1 Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 3028 |====================================== ZFS 64k - QCOW2 64k - NTFS 64k ... 2868 |==================================== ZFS 16k - QCOW2 16k - NTFS 16k ... 2058 |========================== ZFS 4k - QCOW2 4k - NTFS - 4k .... 1460 |================== ZFS 128k - QCOW2 128k - NTFS 64k . 2715 |================================== ZFS 128k - QCOW2 128k - NTFS 4k .. 2917 |===================================== ZFS 128k - QCOW2 64k - NTFS 64k .. 2851 |==================================== ZFS 64k - QCOW2 64K - NTFS 4k .... 2896 |==================================== ZFS 16k - QCOW2 16k - NTFS 4k .... 2002 |========================= ZFS 64k - QCOW2 64k - NTFS 16k ... 2809 |=================================== ZFS 64k - QCOW2 16k - NTFS 16k ... 2460 |=============================== ZFS 64k - QCOW2 4k - NTFS 4k ..... 1423 |================== ZFS 64k - QCOW2 16k - NTFS 4k .... 2195 |============================ Flexible IO Tester 3.1 Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 3225 |=================================== ZFS 64k - QCOW2 64k - NTFS 64k ... 2743 |============================== ZFS 16k - QCOW2 16k - NTFS 16k ... 657 |======= ZFS 4k - QCOW2 4k - NTFS - 4k .... 166 |== ZFS 128k - QCOW2 128k - NTFS 64k . 3503 |====================================== ZFS 128k - QCOW2 128k - NTFS 4k .. 3511 |====================================== ZFS 128k - QCOW2 64k - NTFS 64k .. 3286 |==================================== ZFS 64k - QCOW2 64K - NTFS 4k .... 2695 |============================= ZFS 16k - QCOW2 16k - NTFS 4k .... 641 |======= ZFS 64k - QCOW2 64k - NTFS 16k ... 2406 |========================== ZFS 64k - QCOW2 16k - NTFS 16k ... 2573 |============================ ZFS 64k - QCOW2 4k - NTFS 4k ..... 1236 |============= ZFS 64k - QCOW2 16k - NTFS 4k .... 2641 |============================= Flexible IO Tester 3.1 Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 119.00 |================================ ZFS 64k - QCOW2 64k - NTFS 64k ... 125.00 |================================== ZFS 16k - QCOW2 16k - NTFS 16k ... 82.41 |====================== ZFS 4k - QCOW2 4k - NTFS - 4k .... 42.30 |=========== ZFS 128k - QCOW2 128k - NTFS 64k . 76.97 |===================== ZFS 128k - QCOW2 128k - NTFS 4k .. 76.97 |===================== ZFS 128k - QCOW2 64k - NTFS 64k .. 134.00 |==================================== ZFS 64k - QCOW2 64K - NTFS 4k .... 123.00 |================================= ZFS 16k - QCOW2 16k - NTFS 4k .... 73.90 |==================== ZFS 64k - QCOW2 64k - NTFS 16k ... 119.00 |================================ ZFS 64k - QCOW2 16k - NTFS 16k ... 82.13 |====================== ZFS 64k - QCOW2 4k - NTFS 4k ..... 40.23 |=========== ZFS 64k - QCOW2 16k - NTFS 4k .... 88.98 |======================== Flexible IO Tester 3.1 Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 230.00 |================================= ZFS 64k - QCOW2 64k - NTFS 64k ... 247.00 |==================================== ZFS 16k - QCOW2 16k - NTFS 16k ... 155.00 |====================== ZFS 4k - QCOW2 4k - NTFS - 4k .... 74.46 |=========== ZFS 128k - QCOW2 128k - NTFS 64k . 154.00 |====================== ZFS 128k - QCOW2 128k - NTFS 4k .. 159.00 |======================= ZFS 128k - QCOW2 64k - NTFS 64k .. 249.00 |==================================== ZFS 64k - QCOW2 64K - NTFS 4k .... 233.00 |================================== ZFS 16k - QCOW2 16k - NTFS 4k .... 142.00 |===================== ZFS 64k - QCOW2 64k - NTFS 16k ... 238.00 |================================== ZFS 64k - QCOW2 16k - NTFS 16k ... 165.00 |======================== ZFS 64k - QCOW2 4k - NTFS 4k ..... 79.79 |============ ZFS 64k - QCOW2 16k - NTFS 4k .... 171.00 |========================= Flexible IO Tester 3.1 Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 446 |====================================== ZFS 64k - QCOW2 64k - NTFS 64k ... 447 |====================================== ZFS 16k - QCOW2 16k - NTFS 16k ... 282 |======================== ZFS 4k - QCOW2 4k - NTFS - 4k .... 118 |========== ZFS 128k - QCOW2 128k - NTFS 64k . 299 |========================= ZFS 128k - QCOW2 128k - NTFS 4k .. 321 |=========================== ZFS 128k - QCOW2 64k - NTFS 64k .. 463 |======================================= ZFS 64k - QCOW2 64K - NTFS 4k .... 454 |====================================== ZFS 16k - QCOW2 16k - NTFS 4k .... 275 |======================= ZFS 64k - QCOW2 64k - NTFS 16k ... 437 |===================================== ZFS 64k - QCOW2 16k - NTFS 16k ... 305 |========================== ZFS 64k - QCOW2 4k - NTFS 4k ..... 155 |============= ZFS 64k - QCOW2 16k - NTFS 4k .... 313 |========================== Flexible IO Tester 3.1 Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 1342 |==================================== ZFS 64k - QCOW2 64k - NTFS 64k ... 1354 |===================================== ZFS 16k - QCOW2 16k - NTFS 16k ... 617 |================= ZFS 4k - QCOW2 4k - NTFS - 4k .... 169 |===== ZFS 128k - QCOW2 128k - NTFS 64k . 1177 |================================ ZFS 128k - QCOW2 128k - NTFS 4k .. 1170 |================================ ZFS 128k - QCOW2 64k - NTFS 64k .. 1409 |====================================== ZFS 64k - QCOW2 64K - NTFS 4k .... 1164 |=============================== ZFS 16k - QCOW2 16k - NTFS 4k .... 601 |================ ZFS 64k - QCOW2 64k - NTFS 16k ... 1154 |=============================== ZFS 64k - QCOW2 16k - NTFS 16k ... 904 |======================== ZFS 64k - QCOW2 4k - NTFS 4k ..... 464 |============= ZFS 64k - QCOW2 16k - NTFS 4k .... 945 |========================= Flexible IO Tester 3.1 Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 11700 |================================= ZFS 64k - QCOW2 64k - NTFS 64k ... 12133 |================================== ZFS 16k - QCOW2 16k - NTFS 16k ... 10767 |============================== ZFS 4k - QCOW2 4k - NTFS - 4k .... 5047 |============== ZFS 128k - QCOW2 128k - NTFS 64k . 12025 |================================== ZFS 128k - QCOW2 128k - NTFS 4k .. 12333 |================================== ZFS 128k - QCOW2 64k - NTFS 64k .. 12467 |=================================== ZFS 64k - QCOW2 64K - NTFS 4k .... 13233 |===================================== ZFS 16k - QCOW2 16k - NTFS 4k .... 11300 |================================ ZFS 64k - QCOW2 64k - NTFS 16k ... 12547 |=================================== ZFS 64k - QCOW2 16k - NTFS 16k ... 11900 |================================= ZFS 64k - QCOW2 4k - NTFS 4k ..... 5737 |================ ZFS 64k - QCOW2 16k - NTFS 4k .... 11740 |================================= Flexible IO Tester 3.1 Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 212 |==================================== ZFS 64k - QCOW2 64k - NTFS 64k ... 225 |====================================== ZFS 16k - QCOW2 16k - NTFS 16k ... 229 |======================================= ZFS 4k - QCOW2 4k - NTFS - 4k .... 228 |======================================= ZFS 128k - QCOW2 128k - NTFS 64k . 200 |================================== ZFS 128k - QCOW2 128k - NTFS 4k .. 211 |==================================== ZFS 128k - QCOW2 64k - NTFS 64k .. 214 |==================================== ZFS 64k - QCOW2 64K - NTFS 4k .... 212 |==================================== ZFS 16k - QCOW2 16k - NTFS 4k .... 176 |============================== ZFS 64k - QCOW2 64k - NTFS 16k ... 216 |===================================== ZFS 64k - QCOW2 16k - NTFS 16k ... 213 |==================================== ZFS 64k - QCOW2 4k - NTFS 4k ..... 220 |===================================== ZFS 64k - QCOW2 16k - NTFS 4k .... 208 |=================================== Flexible IO Tester 3.1 Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 426 |==================================== ZFS 64k - QCOW2 64k - NTFS 64k ... 442 |===================================== ZFS 16k - QCOW2 16k - NTFS 16k ... 446 |====================================== ZFS 4k - QCOW2 4k - NTFS - 4k .... 460 |======================================= ZFS 128k - QCOW2 128k - NTFS 64k . 374 |================================ ZFS 128k - QCOW2 128k - NTFS 4k .. 402 |================================== ZFS 128k - QCOW2 64k - NTFS 64k .. 423 |==================================== ZFS 64k - QCOW2 64K - NTFS 4k .... 412 |=================================== ZFS 16k - QCOW2 16k - NTFS 4k .... 364 |=============================== ZFS 64k - QCOW2 64k - NTFS 16k ... 430 |==================================== ZFS 64k - QCOW2 16k - NTFS 16k ... 417 |=================================== ZFS 64k - QCOW2 4k - NTFS 4k ..... 417 |=================================== ZFS 64k - QCOW2 16k - NTFS 4k .... 438 |===================================== Flexible IO Tester 3.1 Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 820 |================================== ZFS 64k - QCOW2 64k - NTFS 64k ... 860 |==================================== ZFS 16k - QCOW2 16k - NTFS 16k ... 865 |==================================== ZFS 4k - QCOW2 4k - NTFS - 4k .... 928 |======================================= ZFS 128k - QCOW2 128k - NTFS 64k . 750 |================================ ZFS 128k - QCOW2 128k - NTFS 4k .. 804 |================================== ZFS 128k - QCOW2 64k - NTFS 64k .. 835 |=================================== ZFS 64k - QCOW2 64K - NTFS 4k .... 810 |================================== ZFS 16k - QCOW2 16k - NTFS 4k .... 694 |============================= ZFS 64k - QCOW2 64k - NTFS 16k ... 839 |=================================== ZFS 64k - QCOW2 16k - NTFS 16k ... 801 |================================== ZFS 64k - QCOW2 4k - NTFS 4k ..... 841 |=================================== ZFS 64k - QCOW2 16k - NTFS 4k .... 822 |=================================== Flexible IO Tester 3.1 Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 2917 |================================== ZFS 64k - QCOW2 64k - NTFS 64k ... 3150 |===================================== ZFS 16k - QCOW2 16k - NTFS 16k ... 2997 |=================================== ZFS 4k - QCOW2 4k - NTFS - 4k .... 3217 |====================================== ZFS 128k - QCOW2 128k - NTFS 64k . 2755 |================================= ZFS 128k - QCOW2 128k - NTFS 4k .. 2906 |================================== ZFS 128k - QCOW2 64k - NTFS 64k .. 2944 |=================================== ZFS 64k - QCOW2 64K - NTFS 4k .... 3028 |==================================== ZFS 16k - QCOW2 16k - NTFS 4k .... 2626 |=============================== ZFS 64k - QCOW2 64k - NTFS 16k ... 2850 |================================== ZFS 64k - QCOW2 16k - NTFS 16k ... 3082 |==================================== ZFS 64k - QCOW2 4k - NTFS 4k ..... 2903 |================================== ZFS 64k - QCOW2 16k - NTFS 4k .... 2893 |================================== Flexible IO Tester 3.1 Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 3393 |====================================== ZFS 64k - QCOW2 64k - NTFS 64k ... 2678 |============================== ZFS 16k - QCOW2 16k - NTFS 16k ... 619 |======= ZFS 4k - QCOW2 4k - NTFS - 4k .... 162 |== ZFS 128k - QCOW2 128k - NTFS 64k . 3147 |=================================== ZFS 128k - QCOW2 128k - NTFS 4k .. 3293 |===================================== ZFS 128k - QCOW2 64k - NTFS 64k .. 3316 |===================================== ZFS 64k - QCOW2 64K - NTFS 4k .... 2574 |============================= ZFS 16k - QCOW2 16k - NTFS 4k .... 612 |======= ZFS 64k - QCOW2 64k - NTFS 16k ... 2502 |============================ ZFS 64k - QCOW2 16k - NTFS 16k ... 2550 |============================= ZFS 64k - QCOW2 4k - NTFS 4k ..... 1548 |================= ZFS 64k - QCOW2 16k - NTFS 4k .... 2533 |============================ Flexible IO Tester 3.1 Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 204.00 |==================================== ZFS 64k - QCOW2 64k - NTFS 64k ... 206.00 |==================================== ZFS 16k - QCOW2 16k - NTFS 16k ... 159.00 |============================ ZFS 4k - QCOW2 4k - NTFS - 4k .... 65.83 |============ ZFS 128k - QCOW2 128k - NTFS 64k . 96.37 |================= ZFS 128k - QCOW2 128k - NTFS 4k .. 92.61 |================ ZFS 128k - QCOW2 64k - NTFS 64k .. 199.00 |=================================== ZFS 64k - QCOW2 64K - NTFS 4k .... 193.00 |================================== ZFS 16k - QCOW2 16k - NTFS 4k .... 130.00 |======================= ZFS 64k - QCOW2 64k - NTFS 16k ... 194.00 |================================== ZFS 64k - QCOW2 16k - NTFS 16k ... 171.00 |============================== ZFS 64k - QCOW2 4k - NTFS 4k ..... 65.73 |=========== ZFS 64k - QCOW2 16k - NTFS 4k .... 167.00 |============================= Flexible IO Tester 3.1 Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 373 |======================================= ZFS 64k - QCOW2 64k - NTFS 64k ... 363 |====================================== ZFS 16k - QCOW2 16k - NTFS 16k ... 256 |=========================== ZFS 4k - QCOW2 4k - NTFS - 4k .... 102 |=========== ZFS 128k - QCOW2 128k - NTFS 64k . 191 |==================== ZFS 128k - QCOW2 128k - NTFS 4k .. 181 |=================== ZFS 128k - QCOW2 64k - NTFS 64k .. 369 |======================================= ZFS 64k - QCOW2 64K - NTFS 4k .... 349 |==================================== ZFS 16k - QCOW2 16k - NTFS 4k .... 223 |======================= ZFS 64k - QCOW2 64k - NTFS 16k ... 325 |================================== ZFS 64k - QCOW2 16k - NTFS 16k ... 303 |================================ ZFS 64k - QCOW2 4k - NTFS 4k ..... 133 |============== ZFS 64k - QCOW2 16k - NTFS 4k .... 291 |============================== Flexible IO Tester 3.1 Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 625 |======================================= ZFS 64k - QCOW2 64k - NTFS 64k ... 610 |====================================== ZFS 16k - QCOW2 16k - NTFS 16k ... 415 |========================== ZFS 4k - QCOW2 4k - NTFS - 4k .... 140 |========= ZFS 128k - QCOW2 128k - NTFS 64k . 378 |======================= ZFS 128k - QCOW2 128k - NTFS 4k .. 381 |======================== ZFS 128k - QCOW2 64k - NTFS 64k .. 628 |======================================= ZFS 64k - QCOW2 64K - NTFS 4k .... 534 |================================= ZFS 16k - QCOW2 16k - NTFS 4k .... 373 |======================= ZFS 64k - QCOW2 64k - NTFS 16k ... 503 |=============================== ZFS 64k - QCOW2 16k - NTFS 16k ... 519 |================================ ZFS 64k - QCOW2 4k - NTFS 4k ..... 231 |============== ZFS 64k - QCOW2 16k - NTFS 4k .... 493 |=============================== Flexible IO Tester 3.1 Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k .. 1360 |=================================== ZFS 64k - QCOW2 64k - NTFS 64k .. 1500 |======================================= ZFS 16k - QCOW2 16k - NTFS 16k .. 585 |=============== ZFS 4k - QCOW2 4k - NTFS - 4k ... 187 |===== ZFS 128k - QCOW2 128k - NTFS 4k . 1153 |============================== ZFS 128k - QCOW2 64k - NTFS 64k . 1486 |======================================= ZFS 64k - QCOW2 64K - NTFS 4k ... 1190 |=============================== ZFS 16k - QCOW2 16k - NTFS 4k ... 601 |================ ZFS 64k - QCOW2 64k - NTFS 16k .. 1152 |============================== ZFS 64k - QCOW2 16k - NTFS 16k .. 1233 |================================ ZFS 64k - QCOW2 4k - NTFS 4k .... 633 |================ ZFS 64k - QCOW2 16k - NTFS 4k ... 1210 |=============================== IOzone 3.465 Record Size: 1MB - File Size: 8GB - Disk Test: Read Performance MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 1867.17 |================================= ZFS 64k - QCOW2 64k - NTFS 64k ... 1852.57 |================================= ZFS 16k - QCOW2 16k - NTFS 16k ... 1492.24 |========================== ZFS 4k - QCOW2 4k - NTFS - 4k .... 699.92 |============ ZFS 128k - QCOW2 128k - NTFS 64k . 1885.86 |================================= ZFS 128k - QCOW2 128k - NTFS 4k .. 1929.75 |================================== ZFS 128k - QCOW2 64k - NTFS 64k .. 1896.74 |================================== ZFS 64k - QCOW2 64K - NTFS 4k .... 1893.33 |================================== ZFS 16k - QCOW2 16k - NTFS 4k .... 1611.63 |============================= ZFS 64k - QCOW2 64k - NTFS 16k ... 1901.82 |================================== ZFS 64k - QCOW2 16k - NTFS 16k ... 1855.03 |================================= ZFS 64k - QCOW2 4k - NTFS 4k ..... 1787.43 |================================ ZFS 64k - QCOW2 16k - NTFS 4k .... 1972.18 |=================================== IOzone 3.465 Record Size: 4Kb - File Size: 8GB - Disk Test: Read Performance MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 561.17 |=================================== ZFS 64k - QCOW2 64k - NTFS 64k ... 563.02 |=================================== ZFS 16k - QCOW2 16k - NTFS 16k ... 437.64 |=========================== ZFS 4k - QCOW2 4k - NTFS - 4k .... 296.75 |================== ZFS 128k - QCOW2 128k - NTFS 64k . 576.96 |==================================== ZFS 128k - QCOW2 128k - NTFS 4k .. 580.74 |==================================== ZFS 128k - QCOW2 64k - NTFS 64k .. 573.83 |==================================== ZFS 64k - QCOW2 64K - NTFS 4k .... 561.94 |=================================== ZFS 16k - QCOW2 16k - NTFS 4k .... 459.38 |============================ ZFS 64k - QCOW2 64k - NTFS 16k ... 551.31 |================================== ZFS 64k - QCOW2 16k - NTFS 16k ... 539.59 |================================= ZFS 64k - QCOW2 4k - NTFS 4k ..... 526.52 |================================= ZFS 64k - QCOW2 16k - NTFS 4k .... 571.72 |=================================== IOzone 3.465 Record Size: 1MB - File Size: 8GB - Disk Test: Write Performance MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 1027.92 |========================== ZFS 64k - QCOW2 64k - NTFS 64k ... 1150.28 |============================== ZFS 16k - QCOW2 16k - NTFS 16k ... 748.10 |=================== ZFS 4k - QCOW2 4k - NTFS - 4k .... 254.89 |======= ZFS 128k - QCOW2 128k - NTFS 64k . 1364.73 |=================================== ZFS 128k - QCOW2 128k - NTFS 4k .. 1016.09 |========================== ZFS 128k - QCOW2 64k - NTFS 64k .. 1326.99 |================================== ZFS 64k - QCOW2 64K - NTFS 4k .... 1043.66 |=========================== ZFS 16k - QCOW2 16k - NTFS 4k .... 606.42 |================ ZFS 64k - QCOW2 64k - NTFS 16k ... 1072.11 |=========================== ZFS 64k - QCOW2 16k - NTFS 16k ... 1133.03 |============================= ZFS 64k - QCOW2 4k - NTFS 4k ..... 925.75 |======================== ZFS 64k - QCOW2 16k - NTFS 4k .... 1007.22 |========================== IOzone 3.465 Record Size: 4Kb - File Size: 8GB - Disk Test: Write Performance MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 439.51 |======================== ZFS 64k - QCOW2 64k - NTFS 64k ... 622.91 |=================================== ZFS 16k - QCOW2 16k - NTFS 16k ... 565.58 |=============================== ZFS 4k - QCOW2 4k - NTFS - 4k .... 243.93 |============== ZFS 128k - QCOW2 128k - NTFS 64k . 648.44 |==================================== ZFS 128k - QCOW2 128k - NTFS 4k .. 446.27 |========================= ZFS 128k - QCOW2 64k - NTFS 64k .. 615.63 |================================== ZFS 64k - QCOW2 64K - NTFS 4k .... 395.38 |====================== ZFS 16k - QCOW2 16k - NTFS 4k .... 396.63 |====================== ZFS 64k - QCOW2 64k - NTFS 16k ... 552.19 |=============================== ZFS 64k - QCOW2 16k - NTFS 16k ... 542.85 |============================== ZFS 64k - QCOW2 4k - NTFS 4k ..... 426.39 |======================== ZFS 64k - QCOW2 16k - NTFS 4k .... 485.37 |=========================== IOzone 3.465 Record Size: 64Kb - File Size: 8GB - Disk Test: Read Performance MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 807.93 |================================= ZFS 64k - QCOW2 64k - NTFS 64k ... 833.63 |================================== ZFS 16k - QCOW2 16k - NTFS 16k ... 655.99 |=========================== ZFS 4k - QCOW2 4k - NTFS - 4k .... 408.54 |================= ZFS 128k - QCOW2 128k - NTFS 64k . 875.13 |==================================== ZFS 128k - QCOW2 128k - NTFS 4k .. 835.14 |================================== ZFS 128k - QCOW2 64k - NTFS 64k .. 826.00 |================================== ZFS 64k - QCOW2 64K - NTFS 4k .... 792.23 |================================= ZFS 16k - QCOW2 16k - NTFS 4k .... 644.41 |=========================== ZFS 64k - QCOW2 64k - NTFS 16k ... 810.39 |================================= ZFS 64k - QCOW2 16k - NTFS 16k ... 806.79 |================================= ZFS 64k - QCOW2 4k - NTFS 4k ..... 799.85 |================================= ZFS 64k - QCOW2 16k - NTFS 4k .... 786.52 |================================ IOzone 3.465 Record Size: 64Kb - File Size: 8GB - Disk Test: Write Performance MB/s > Higher Is Better ZFS 128k - QCOW2 64k - NTFS 4k ... 947.01 |========================= ZFS 64k - QCOW2 64k - NTFS 64k ... 1154.15 |=============================== ZFS 16k - QCOW2 16k - NTFS 16k ... 743.78 |==================== ZFS 4k - QCOW2 4k - NTFS - 4k .... 266.34 |======= ZFS 128k - QCOW2 128k - NTFS 64k . 1293.46 |================================== ZFS 128k - QCOW2 128k - NTFS 4k .. 951.01 |========================= ZFS 128k - QCOW2 64k - NTFS 64k .. 1318.79 |=================================== ZFS 64k - QCOW2 64K - NTFS 4k .... 808.15 |===================== ZFS 16k - QCOW2 16k - NTFS 4k .... 583.25 |=============== ZFS 64k - QCOW2 64k - NTFS 16k ... 1125.91 |============================== ZFS 64k - QCOW2 16k - NTFS 16k ... 1089.60 |============================= ZFS 64k - QCOW2 4k - NTFS 4k ..... 908.61 |======================== ZFS 64k - QCOW2 16k - NTFS 4k .... 977.73 |==========================