ZFS + KVM + NTFS Benchmarking Various Record Sizes

This benchmark aims to identify optimal combination of record sizes (i.e., recordsize for ZFS, cluster_size for QCOW2, allocation unit size for NTFS) for a Windows 10 Guest VM running on top of ZFS + KVM. Host: Dual Intel Xeon 6130 @ 2.10GHz, 384GB RAM, CentOS 7.6 Virtualization Base Environment, ZFS 0.8.0 via EPEL zfs-testing repository ZFS Config: 4 x RAIDZ2 (5 x 8TB 7.2K HDD) + 2 x OPTANE 900 280GB SLOG MIRROR + OPTANE 900 480GB CACHE Windows Guest: 2 vCPUs, 4GB RAM, 40 GB QCOW2 (bus:virtio, cache:writeback, io:default, cluster size 4k ~ 64k)

HTML result view exported from: https://openbenchmarking.org/result/1906108-HV-MERGE807489.

ZFS + KVM + NTFS Benchmarking Various Record SizesProcessorMotherboardMemoryDiskGraphicsOSKernelDisplay DriverFile-SystemScreen ResolutionZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k2 x Intel Xeon (Skylake IBRS) @ 2.10GHz (2 Cores)(0.5.1 BIOS)1 x 4096 MB40GB Red Hat VirtIOMicrosoft Basic DisplayMicrosoft Windows 10 Enterprise Evaluation Build 1836210.0 (x86_64)10.0.18362.1NTFS1024x768OpenBenchmarking.org

ZFS + KVM + NTFS Benchmarking Various Record Sizesfio: Rand Read - Windows AIO - No - Yes - 1MBfio: Rand Read - Windows AIO - No - Yes - 4KBfio: Rand Read - Windows AIO - No - Yes - 8KBfio: Rand Read - Windows AIO - No - Yes - 16KBfio: Rand Read - Windows AIO - No - Yes - 64KBfio: Rand Write - Windows AIO - No - Yes - 1MBfio: Rand Write - Windows AIO - No - Yes - 4KBfio: Rand Write - Windows AIO - No - Yes - 8KBfio: Rand Write - Windows AIO - No - Yes - 16KBfio: Rand Write - Windows AIO - No - Yes - 64KBfio: Seq Read - Windows AIO - No - Yes - 1MBfio: Seq Read - Windows AIO - No - Yes - 4KBfio: Seq Read - Windows AIO - No - Yes - 8KBfio: Seq Read - Windows AIO - No - Yes - 16KBfio: Seq Read - Windows AIO - No - Yes - 64KBfio: Seq Write - Windows AIO - No - Yes - 1MBfio: Seq Write - Windows AIO - No - Yes - 4KBfio: Seq Write - Windows AIO - No - Yes - 8KBfio: Seq Write - Windows AIO - No - Yes - 16KBfio: Seq Write - Windows AIO - No - Yes - 64KBiozone: 1MB - 8GB - Read Performanceiozone: 4Kb - 8GB - Read Performanceiozone: 1MB - 8GB - Write Performanceiozone: 4Kb - 8GB - Write Performanceiozone: 64Kb - 8GB - Read Performanceiozone: 64Kb - 8GB - Write PerformanceZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k11500234463831286827431252474471354121332254428603150267820636361015001852.57563.021150.28622.91833.631154.157675146288578205865782.411552826171076722944686529976191592564155851492.24437.64748.10565.58655.99743.784191104.47198433146016642.3074.461181695047228460928321716265.83102140187699.92296.75254.89243.93408.54266.3412875211408791289626951232334541164132332124128103028257419334953411901893.33561.941043.66395.38792.23808.159830126252500200264173.901422756011130017636469426266121302233736011611.63459.38606.42396.63644.41583.2512853211420830280924061192384371154125472164308392850250219432550311521901.82551.311072.11552.19810.391125.9198591603246402460257382.13165305904119002134178013082255017130351912331855.03539.591133.03542.85806.791089.6052701102283971423123640.2379.7915546457372204178412903154865.731332316331787.43526.52925.75426.39799.85908.6190071532945922195264188.98171313945117402084388222893253316729149312101972.18571.721007.22485.37786.52977.73OpenBenchmarking.org

Flexible IO Tester

Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 1MB

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 1MBZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k3K6K9K12K15KSE +/- 210.22, N = 15SE +/- 106.26, N = 4SE +/- 37.82, N = 3SE +/- 130.59, N = 8SE +/- 75.55, N = 3SE +/- 139.00, N = 15SE +/- 108.50, N = 3SE +/- 41.46, N = 3SE +/- 59.54, N = 3115007675419112875983012853985952709007

Flexible IO Tester

Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 4KB

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 4KBZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k50100150200250SE +/- 3.84, N = 3SE +/- 2.03, N = 3SE +/- 1.04, N = 15SE +/- 2.33, N = 3SE +/- 1.24, N = 15SE +/- 2.96, N = 15SE +/- 2.52, N = 15SE +/- 1.00, N = 3SE +/- 2.19, N = 3234.00146.00104.47211.00126.00211.00160.00110.00153.00

Flexible IO Tester

Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 8KB

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 8KBZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k100200300400500SE +/- 7.21, N = 15SE +/- 2.60, N = 3SE +/- 2.74, N = 4SE +/- 3.06, N = 3SE +/- 3.79, N = 3SE +/- 5.09, N = 15SE +/- 3.89, N = 6SE +/- 1.20, N = 3SE +/- 2.45, N = 15463288198408252420324228294

Flexible IO Tester

Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 16KB

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 16KBZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k2004006008001000SE +/- 11.79, N = 3SE +/- 5.76, N = 15SE +/- 4.95, N = 15SE +/- 9.43, N = 15SE +/- 5.49, N = 15SE +/- 8.01, N = 15SE +/- 7.55, N = 3SE +/- 4.97, N = 15SE +/- 5.24, N = 3831578433791500830640397592

Flexible IO Tester

Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 64KB

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 64KBZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k6001200180024003000SE +/- 30.89, N = 15SE +/- 21.48, N = 15SE +/- 17.41, N = 15SE +/- 29.53, N = 8SE +/- 23.87, N = 15SE +/- 34.30, N = 5SE +/- 21.63, N = 13SE +/- 20.58, N = 3SE +/- 26.34, N = 15286820581460289620022809246014232195

Flexible IO Tester

Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 1MB

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 1MBZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k6001200180024003000SE +/- 57.14, N = 15SE +/- 13.92, N = 15SE +/- 5.04, N = 15SE +/- 28.67, N = 3SE +/- 25.74, N = 15SE +/- 78.39, N = 15SE +/- 69.43, N = 15SE +/- 22.28, N = 15SE +/- 76.03, N = 15274365716626956412406257312362641

Flexible IO Tester

Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 4KB

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 4KBZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k306090120150SE +/- 1.41, N = 15SE +/- 0.95, N = 15SE +/- 0.25, N = 3SE +/- 1.76, N = 3SE +/- 1.04, N = 3SE +/- 0.71, N = 15SE +/- 0.15, N = 3SE +/- 1.41, N = 15125.0082.4142.30123.0073.90119.0082.1340.2388.98

Flexible IO Tester

Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 8KB

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 8KBZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k50100150200250SE +/- 3.79, N = 3SE +/- 1.46, N = 15SE +/- 1.40, N = 12SE +/- 3.01, N = 15SE +/- 1.76, N = 3SE +/- 2.40, N = 3SE +/- 0.63, N = 14247.00155.0074.46233.00142.00238.00165.0079.79171.00

Flexible IO Tester

Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 16KB

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 16KBZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k100200300400500SE +/- 4.37, N = 3SE +/- 3.37, N = 15SE +/- 1.39, N = 15SE +/- 3.71, N = 3SE +/- 4.19, N = 9SE +/- 4.98, N = 3SE +/- 0.88, N = 3SE +/- 2.08, N = 3447282118454275437305155313

Flexible IO Tester

Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 64KB

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 64KBZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k30060090012001500SE +/- 28.55, N = 15SE +/- 21.03, N = 15SE +/- 4.42, N = 15SE +/- 27.25, N = 15SE +/- 25.28, N = 15SE +/- 14.83, N = 5SE +/- 9.87, N = 3SE +/- 2.60, N = 3SE +/- 13.06, N = 15135461716911646011154904464945

Flexible IO Tester

Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 1MB

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 1MBZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k3K6K9K12K15KSE +/- 88.19, N = 3SE +/- 88.19, N = 3SE +/- 22.91, N = 3SE +/- 115.33, N = 15SE +/- 115.47, N = 3SE +/- 107.29, N = 15SE +/- 57.74, N = 3SE +/- 51.66, N = 10SE +/- 143.53, N = 51213310767504713233113001254711900573711740

Flexible IO Tester

Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 4KB

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 4KBZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k50100150200250SE +/- 3.12, N = 15SE +/- 2.19, N = 3SE +/- 2.30, N = 15SE +/- 1.20, N = 3SE +/- 2.04, N = 15SE +/- 1.45, N = 3SE +/- 2.60, N = 3SE +/- 2.40, N = 3225229228212176216213220208

Flexible IO Tester

Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 8KB

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 8KBZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k100200300400500SE +/- 5.71, N = 15SE +/- 4.81, N = 3SE +/- 5.80, N = 5SE +/- 1.76, N = 3SE +/- 3.78, N = 15SE +/- 2.33, N = 3SE +/- 6.64, N = 3SE +/- 6.84, N = 3SE +/- 1.53, N = 3442446460412364430417417438

Flexible IO Tester

Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 16KB

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 16KBZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k2004006008001000SE +/- 10.12, N = 15SE +/- 7.51, N = 3SE +/- 7.69, N = 3SE +/- 12.37, N = 15SE +/- 3.93, N = 3SE +/- 9.13, N = 3SE +/- 7.36, N = 3SE +/- 10.56, N = 15SE +/- 5.90, N = 3860865928810694839801841822

Flexible IO Tester

Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 64KB

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 64KBZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k7001400210028003500SE +/- 43.44, N = 4SE +/- 24.67, N = 3SE +/- 32.24, N = 8SE +/- 7.45, N = 3SE +/- 34.35, N = 3SE +/- 37.55, N = 3SE +/- 10.33, N = 3SE +/- 24.31, N = 15SE +/- 13.65, N = 3315029973217302826262850308229032893

Flexible IO Tester

Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 1MB

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 1MBZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k6001200180024003000SE +/- 49.81, N = 15SE +/- 19.74, N = 15SE +/- 4.19, N = 15SE +/- 26.21, N = 3SE +/- 20.61, N = 15SE +/- 71.45, N = 15SE +/- 65.35, N = 12SE +/- 27.56, N = 15SE +/- 122.35, N = 12267861916225746122502255015482533

Flexible IO Tester

Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 4KB

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 4KBZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k50100150200250SE +/- 2.72, N = 4SE +/- 0.76, N = 15SE +/- 2.60, N = 3SE +/- 2.19, N = 3SE +/- 2.91, N = 3SE +/- 2.09, N = 15SE +/- 0.95, N = 3SE +/- 2.52, N = 3206.00159.0065.83193.00130.00194.00171.0065.73167.00

Flexible IO Tester

Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 8KB

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 8KBZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k80160240320400SE +/- 2.00, N = 3SE +/- 2.08, N = 3SE +/- 1.20, N = 3SE +/- 4.16, N = 3SE +/- 2.66, N = 15SE +/- 4.59, N = 4SE +/- 2.76, N = 15SE +/- 1.00, N = 3SE +/- 4.08, N = 15363256102349223325303133291

Flexible IO Tester

Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 16KB

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 16KBZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k130260390520650SE +/- 8.69, N = 3SE +/- 3.61, N = 3SE +/- 2.12, N = 15SE +/- 6.12, N = 6SE +/- 6.17, N = 3SE +/- 5.46, N = 3SE +/- 4.71, N = 15SE +/- 3.04, N = 5SE +/- 7.69, N = 3610415140534373503519231493

Flexible IO Tester

Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 64KB

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 64KBZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k30060090012001500SE +/- 29.00, N = 15SE +/- 18.13, N = 12SE +/- 5.51, N = 12SE +/- 18.33, N = 3SE +/- 9.40, N = 3SE +/- 37.86, N = 12SE +/- 25.11, N = 15SE +/- 5.91, N = 15SE +/- 14.28, N = 1515005851871190601115212336331210

IOzone

Record Size: 1MB - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 8GB - Disk Test: Read PerformanceZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k400800120016002000SE +/- 28.21, N = 3SE +/- 19.59, N = 15SE +/- 8.14, N = 3SE +/- 23.36, N = 5SE +/- 7.34, N = 3SE +/- 32.92, N = 15SE +/- 19.66, N = 15SE +/- 24.78, N = 3SE +/- 22.67, N = 31852.571492.24699.921893.331611.631901.821855.031787.431972.18

IOzone

Record Size: 4Kb - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 8GB - Disk Test: Read PerformanceZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k120240360480600SE +/- 1.57, N = 3SE +/- 3.62, N = 3SE +/- 5.26, N = 12SE +/- 6.39, N = 3SE +/- 5.81, N = 5SE +/- 6.80, N = 5SE +/- 4.68, N = 3SE +/- 6.81, N = 3SE +/- 5.19, N = 15563.02437.64296.75561.94459.38551.31539.59526.52571.72

IOzone

Record Size: 1MB - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 8GB - Disk Test: Write PerformanceZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k2004006008001000SE +/- 14.89, N = 15SE +/- 7.98, N = 15SE +/- 1.51, N = 3SE +/- 29.58, N = 12SE +/- 4.23, N = 3SE +/- 37.94, N = 15SE +/- 20.17, N = 15SE +/- 13.11, N = 15SE +/- 29.59, N = 151150.28748.10254.891043.66606.421072.111133.03925.751007.22

IOzone

Record Size: 4Kb - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 8GB - Disk Test: Write PerformanceZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k130260390520650SE +/- 7.97, N = 3SE +/- 2.28, N = 3SE +/- 3.44, N = 3SE +/- 8.83, N = 15SE +/- 4.57, N = 3SE +/- 8.87, N = 15SE +/- 7.62, N = 15SE +/- 1.76, N = 3SE +/- 10.50, N = 15622.91565.58243.93395.38396.63552.19542.85426.39485.37

IOzone

Record Size: 64Kb - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 8GB - Disk Test: Read PerformanceZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k2004006008001000SE +/- 11.05, N = 3SE +/- 6.93, N = 3SE +/- 4.64, N = 15SE +/- 6.94, N = 11SE +/- 6.68, N = 3SE +/- 9.09, N = 15SE +/- 5.12, N = 3SE +/- 7.14, N = 3SE +/- 10.06, N = 4833.63655.99408.54792.23644.41810.39806.79799.85786.52

IOzone

Record Size: 64Kb - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 8GB - Disk Test: Write PerformanceZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k2004006008001000SE +/- 13.91, N = 15SE +/- 11.06, N = 15SE +/- 3.55, N = 4SE +/- 6.36, N = 3SE +/- 3.74, N = 3SE +/- 13.45, N = 5SE +/- 14.52, N = 5SE +/- 23.22, N = 13SE +/- 29.69, N = 151154.15743.78266.34808.15583.251125.911089.60908.61977.73

Phoronix Test Suite v10.8.4