ZFS + KVM + NTFS Benchmarking Various Record Sizes

This benchmark aims to identify optimal combination of record sizes (i.e., recordsize for ZFS, cluster_size for QCOW2, allocation unit size for NTFS) for a Windows 10 Guest VM running on top of ZFS + KVM. Host: Dual Intel Xeon 6130 @ 2.10GHz, 384GB RAM, CentOS 7.6 Virtualization Base Environment, ZFS 0.8.0 via EPEL zfs-testing repository ZFS Config: 4 x RAIDZ2 (5 x 8TB 7.2K HDD) + 2 x OPTANE 900 280GB SLOG MIRROR + OPTANE 900 480GB CACHE Windows Guest: 2 vCPUs, 4GB RAM, 40 GB QCOW2 (bus:virtio, cache:writeback, io:default, cluster size 4k ~ 128k)

HTML result view exported from: https://openbenchmarking.org/result/1906129-HV-MERGE333894.

ZFS + KVM + NTFS Benchmarking Various Record SizesProcessorMotherboardMemoryDiskGraphicsOSKernelDisplay DriverFile-SystemScreen ResolutionZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k2 x Intel Xeon (Skylake IBRS) @ 2.10GHz (2 Cores)(0.5.1 BIOS)1 x 4096 MB40GB Red Hat VirtIOMicrosoft Basic DisplayMicrosoft Windows 10 Enterprise Evaluation Build 1836210.0 (x86_64)10.0.18362.1NTFS1024x768OpenBenchmarking.org

ZFS + KVM + NTFS Benchmarking Various Record Sizesfio: Rand Read - Windows AIO - No - Yes - 1MB - Default Test Directoryfio: Rand Read - Windows AIO - No - Yes - 4KB - Default Test Directoryfio: Rand Read - Windows AIO - No - Yes - 8KB - Default Test Directoryfio: Rand Read - Windows AIO - No - Yes - 16KB - Default Test Directoryfio: Rand Read - Windows AIO - No - Yes - 64KB - Default Test Directoryfio: Rand Write - Windows AIO - No - Yes - 1MB - Default Test Directoryfio: Rand Write - Windows AIO - No - Yes - 4KB - Default Test Directoryfio: Rand Write - Windows AIO - No - Yes - 8KB - Default Test Directoryfio: Rand Write - Windows AIO - No - Yes - 16KB - Default Test Directoryfio: Rand Write - Windows AIO - No - Yes - 64KB - Default Test Directoryfio: Seq Read - Windows AIO - No - Yes - 1MB - Default Test Directoryfio: Seq Read - Windows AIO - No - Yes - 4KB - Default Test Directoryfio: Seq Read - Windows AIO - No - Yes - 8KB - Default Test Directoryfio: Seq Read - Windows AIO - No - Yes - 16KB - Default Test Directoryfio: Seq Read - Windows AIO - No - Yes - 64KB - Default Test Directoryfio: Seq Write - Windows AIO - No - Yes - 1MB - Default Test Directoryfio: Seq Write - Windows AIO - No - Yes - 4KB - Default Test Directoryfio: Seq Write - Windows AIO - No - Yes - 8KB - Default Test Directoryfio: Seq Write - Windows AIO - No - Yes - 16KB - Default Test Directoryfio: Seq Write - Windows AIO - No - Yes - 64KB - Default Test Directoryiozone: 1MB - 8GB - Read Performanceiozone: 4Kb - 8GB - Read Performanceiozone: 1MB - 8GB - Write Performanceiozone: 4Kb - 8GB - Write Performanceiozone: 64Kb - 8GB - Read Performanceiozone: 64Kb - 8GB - Write PerformanceZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k12500204416838302832251192304461342117002124268202917339320437362513601867.17561.171027.92439.51807.93947.0111500234463831286827431252474471354121332254428603150267820636361015001852.57563.021150.28622.91833.631154.157675146288578205865782.411552826171076722944686529976191592564155851492.24437.64748.10565.58655.99743.784191104.47198433146016642.3074.461181695047228460928321716265.83102140187699.92296.75254.89243.93408.54266.34121602054088082715350376.971542991177120252003747502755314796.371913781885.86576.961364.73648.44875.131293.46125332094168282917351176.971593211170123332114028042906329392.6118138111531929.75580.741016.09446.27835.14951.0112554208410815285132861342494631409124672144238352944331619936962814861896.74573.831326.99615.63826.001318.7912875211408791289626951232334541164132332124128103028257419334953411901893.33561.941043.66395.38792.23808.159830126252500200264173.901422756011130017636469426266121302233736011611.63459.38606.42396.63644.41583.2512853211420830280924061192384371154125472164308392850250219432550311521901.82551.311072.11552.19810.391125.9198591603246402460257382.13165305904119002134178013082255017130351912331855.03539.591133.03542.85806.791089.6052701102283971423123640.2379.7915546457372204178412903154865.731332316331787.43526.52925.75426.39799.85908.6190071532945922195264188.98171313945117402084388222893253316729149312101972.18571.721007.22485.37786.52977.73OpenBenchmarking.org

Flexible IO Tester

Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test DirectoryZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k3K6K9K12K15KSE +/- 173.21, N = 3SE +/- 210.22, N = 15SE +/- 106.26, N = 4SE +/- 37.82, N = 3SE +/- 162.36, N = 15SE +/- 145.30, N = 3SE +/- 143.94, N = 13SE +/- 130.59, N = 8SE +/- 75.55, N = 3SE +/- 139.00, N = 15SE +/- 108.50, N = 3SE +/- 41.46, N = 3SE +/- 59.54, N = 312500115007675419112160125331255412875983012853985952709007

Flexible IO Tester

Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k50100150200250SE +/- 3.90, N = 14SE +/- 3.84, N = 3SE +/- 2.03, N = 3SE +/- 1.04, N = 15SE +/- 1.76, N = 15SE +/- 1.45, N = 3SE +/- 2.24, N = 7SE +/- 2.33, N = 3SE +/- 1.24, N = 15SE +/- 2.96, N = 15SE +/- 2.52, N = 15SE +/- 1.00, N = 3SE +/- 2.19, N = 3204.00234.00146.00104.47205.00209.00208.00211.00126.00211.00160.00110.00153.00

Flexible IO Tester

Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test DirectoryZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k100200300400500SE +/- 5.30, N = 15SE +/- 7.21, N = 15SE +/- 2.60, N = 3SE +/- 2.74, N = 4SE +/- 4.51, N = 3SE +/- 6.56, N = 3SE +/- 2.85, N = 3SE +/- 3.06, N = 3SE +/- 3.79, N = 3SE +/- 5.09, N = 15SE +/- 3.89, N = 6SE +/- 1.20, N = 3SE +/- 2.45, N = 15416463288198408416410408252420324228294

Flexible IO Tester

Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: Default Test DirectoryZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k2004006008001000SE +/- 11.86, N = 3SE +/- 11.79, N = 3SE +/- 5.76, N = 15SE +/- 4.95, N = 15SE +/- 11.41, N = 15SE +/- 9.24, N = 15SE +/- 9.68, N = 3SE +/- 9.43, N = 15SE +/- 5.49, N = 15SE +/- 8.01, N = 15SE +/- 7.55, N = 3SE +/- 4.97, N = 15SE +/- 5.24, N = 3838831578433808828815791500830640397592

Flexible IO Tester

Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test DirectoryZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k6001200180024003000SE +/- 42.72, N = 3SE +/- 30.89, N = 15SE +/- 21.48, N = 15SE +/- 17.41, N = 15SE +/- 26.54, N = 3SE +/- 47.81, N = 3SE +/- 38.74, N = 3SE +/- 29.53, N = 8SE +/- 23.87, N = 15SE +/- 34.30, N = 5SE +/- 21.63, N = 13SE +/- 20.58, N = 3SE +/- 26.34, N = 153028286820581460271529172851289620022809246014232195

Flexible IO Tester

Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test DirectoryZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k8001600240032004000SE +/- 98.84, N = 12SE +/- 57.14, N = 15SE +/- 13.92, N = 15SE +/- 5.04, N = 15SE +/- 28.62, N = 3SE +/- 68.97, N = 14SE +/- 81.14, N = 15SE +/- 28.67, N = 3SE +/- 25.74, N = 15SE +/- 78.39, N = 15SE +/- 69.43, N = 15SE +/- 22.28, N = 15SE +/- 76.03, N = 153225274365716635033511328626956412406257312362641

Flexible IO Tester

Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k306090120150SE +/- 2.03, N = 3SE +/- 1.41, N = 15SE +/- 0.95, N = 15SE +/- 0.25, N = 3SE +/- 0.34, N = 3SE +/- 0.83, N = 15SE +/- 0.88, N = 3SE +/- 1.76, N = 3SE +/- 1.04, N = 3SE +/- 0.71, N = 15SE +/- 0.15, N = 3SE +/- 1.41, N = 15119.00125.0082.4142.3076.9776.97134.00123.0073.90119.0082.1340.2388.98

Flexible IO Tester

Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test DirectoryZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k50100150200250SE +/- 2.08, N = 15SE +/- 3.79, N = 3SE +/- 1.46, N = 15SE +/- 1.40, N = 12SE +/- 3.55, N = 15SE +/- 2.24, N = 15SE +/- 3.01, N = 15SE +/- 1.76, N = 3SE +/- 2.40, N = 3SE +/- 0.63, N = 14230.00247.00155.0074.46154.00159.00249.00233.00142.00238.00165.0079.79171.00

Flexible IO Tester

Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: Default Test DirectoryZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k100200300400500SE +/- 4.13, N = 15SE +/- 4.37, N = 3SE +/- 3.37, N = 15SE +/- 1.39, N = 15SE +/- 2.08, N = 3SE +/- 3.26, N = 15SE +/- 6.71, N = 15SE +/- 3.71, N = 3SE +/- 4.19, N = 9SE +/- 4.98, N = 3SE +/- 0.88, N = 3SE +/- 2.08, N = 3446447282118299321463454275437305155313

Flexible IO Tester

Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test DirectoryZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k30060090012001500SE +/- 16.17, N = 3SE +/- 28.55, N = 15SE +/- 21.03, N = 15SE +/- 4.42, N = 15SE +/- 16.82, N = 15SE +/- 22.57, N = 12SE +/- 36.81, N = 15SE +/- 27.25, N = 15SE +/- 25.28, N = 15SE +/- 14.83, N = 5SE +/- 9.87, N = 3SE +/- 2.60, N = 3SE +/- 13.06, N = 151342135461716911771170140911646011154904464945

Flexible IO Tester

Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test DirectoryZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k3K6K9K12K15KSE +/- 88.19, N = 3SE +/- 88.19, N = 3SE +/- 22.91, N = 3SE +/- 179.70, N = 4SE +/- 136.51, N = 15SE +/- 176.38, N = 3SE +/- 115.33, N = 15SE +/- 115.47, N = 3SE +/- 107.29, N = 15SE +/- 57.74, N = 3SE +/- 51.66, N = 10SE +/- 143.53, N = 5117001213310767504712025123331246713233113001254711900573711740

Flexible IO Tester

Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k50100150200250SE +/- 2.52, N = 3SE +/- 3.12, N = 15SE +/- 2.19, N = 3SE +/- 2.30, N = 15SE +/- 1.66, N = 13SE +/- 2.71, N = 4SE +/- 2.67, N = 3SE +/- 1.20, N = 3SE +/- 2.04, N = 15SE +/- 1.45, N = 3SE +/- 2.60, N = 3SE +/- 2.40, N = 3212225229228200211214212176216213220208

Flexible IO Tester

Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test DirectoryZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k100200300400500SE +/- 1.76, N = 3SE +/- 5.71, N = 15SE +/- 4.81, N = 3SE +/- 5.80, N = 5SE +/- 6.01, N = 3SE +/- 5.21, N = 3SE +/- 3.76, N = 3SE +/- 1.76, N = 3SE +/- 3.78, N = 15SE +/- 2.33, N = 3SE +/- 6.64, N = 3SE +/- 6.84, N = 3SE +/- 1.53, N = 3426442446460374402423412364430417417438

Flexible IO Tester

Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: Default Test DirectoryZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k2004006008001000SE +/- 4.73, N = 3SE +/- 10.12, N = 15SE +/- 7.51, N = 3SE +/- 7.69, N = 3SE +/- 5.70, N = 3SE +/- 6.33, N = 3SE +/- 7.70, N = 15SE +/- 12.37, N = 15SE +/- 3.93, N = 3SE +/- 9.13, N = 3SE +/- 7.36, N = 3SE +/- 10.56, N = 15SE +/- 5.90, N = 3820860865928750804835810694839801841822

Flexible IO Tester

Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test DirectoryZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k7001400210028003500SE +/- 38.07, N = 4SE +/- 43.44, N = 4SE +/- 24.67, N = 3SE +/- 32.24, N = 8SE +/- 8.50, N = 3SE +/- 10.58, N = 3SE +/- 34.87, N = 15SE +/- 7.45, N = 3SE +/- 34.35, N = 3SE +/- 37.55, N = 3SE +/- 10.33, N = 3SE +/- 24.31, N = 15SE +/- 13.65, N = 32917315029973217275529062944302826262850308229032893

Flexible IO Tester

Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 1MB - Disk Target: Default Test DirectoryZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k7001400210028003500SE +/- 57.43, N = 15SE +/- 49.81, N = 15SE +/- 19.74, N = 15SE +/- 4.19, N = 15SE +/- 53.58, N = 15SE +/- 57.13, N = 15SE +/- 80.74, N = 15SE +/- 26.21, N = 3SE +/- 20.61, N = 15SE +/- 71.45, N = 15SE +/- 65.35, N = 12SE +/- 27.56, N = 15SE +/- 122.35, N = 123393267861916231473293331625746122502255015482533

Flexible IO Tester

Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k50100150200250SE +/- 2.36, N = 15SE +/- 2.72, N = 4SE +/- 0.76, N = 15SE +/- 1.00, N = 15SE +/- 1.06, N = 15SE +/- 2.60, N = 3SE +/- 2.19, N = 3SE +/- 2.91, N = 3SE +/- 2.09, N = 15SE +/- 0.95, N = 3SE +/- 2.52, N = 3204.00206.00159.0065.8396.3792.61199.00193.00130.00194.00171.0065.73167.00

Flexible IO Tester

Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 8KB - Disk Target: Default Test DirectoryZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k80160240320400SE +/- 3.51, N = 15SE +/- 2.00, N = 3SE +/- 2.08, N = 3SE +/- 1.20, N = 3SE +/- 3.70, N = 12SE +/- 2.91, N = 15SE +/- 4.16, N = 3SE +/- 2.66, N = 15SE +/- 4.59, N = 4SE +/- 2.76, N = 15SE +/- 1.00, N = 3SE +/- 4.08, N = 15373363256102191181369349223325303133291

Flexible IO Tester

Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: Default Test DirectoryZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k140280420560700SE +/- 10.40, N = 3SE +/- 8.69, N = 3SE +/- 3.61, N = 3SE +/- 2.12, N = 15SE +/- 4.18, N = 15SE +/- 5.36, N = 3SE +/- 7.11, N = 15SE +/- 6.12, N = 6SE +/- 6.17, N = 3SE +/- 5.46, N = 3SE +/- 4.71, N = 15SE +/- 3.04, N = 5SE +/- 7.69, N = 3625610415140378381628534373503519231493

Flexible IO Tester

Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Windows AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: Default Test DirectoryZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k30060090012001500SE +/- 17.28, N = 15SE +/- 29.00, N = 15SE +/- 18.13, N = 12SE +/- 5.51, N = 12SE +/- 12.77, N = 15SE +/- 17.89, N = 15SE +/- 18.33, N = 3SE +/- 9.40, N = 3SE +/- 37.86, N = 12SE +/- 25.11, N = 15SE +/- 5.91, N = 15SE +/- 14.28, N = 1513601500585187115314861190601115212336331210

IOzone

Record Size: 1MB - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 8GB - Disk Test: Read PerformanceZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k400800120016002000SE +/- 39.76, N = 15SE +/- 28.21, N = 3SE +/- 19.59, N = 15SE +/- 8.14, N = 3SE +/- 38.22, N = 15SE +/- 33.35, N = 15SE +/- 31.29, N = 3SE +/- 23.36, N = 5SE +/- 7.34, N = 3SE +/- 32.92, N = 15SE +/- 19.66, N = 15SE +/- 24.78, N = 3SE +/- 22.67, N = 31867.171852.571492.24699.921885.861929.751896.741893.331611.631901.821855.031787.431972.18

IOzone

Record Size: 4Kb - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 8GB - Disk Test: Read PerformanceZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k130260390520650SE +/- 8.16, N = 3SE +/- 1.57, N = 3SE +/- 3.62, N = 3SE +/- 5.26, N = 12SE +/- 6.91, N = 3SE +/- 8.65, N = 4SE +/- 8.49, N = 3SE +/- 6.39, N = 3SE +/- 5.81, N = 5SE +/- 6.80, N = 5SE +/- 4.68, N = 3SE +/- 6.81, N = 3SE +/- 5.19, N = 15561.17563.02437.64296.75576.96580.74573.83561.94459.38551.31539.59526.52571.72

IOzone

Record Size: 1MB - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 8GB - Disk Test: Write PerformanceZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k30060090012001500SE +/- 28.65, N = 15SE +/- 14.89, N = 15SE +/- 7.98, N = 15SE +/- 1.51, N = 3SE +/- 12.35, N = 10SE +/- 27.59, N = 14SE +/- 11.40, N = 3SE +/- 29.58, N = 12SE +/- 4.23, N = 3SE +/- 37.94, N = 15SE +/- 20.17, N = 15SE +/- 13.11, N = 15SE +/- 29.59, N = 151027.921150.28748.10254.891364.731016.091326.991043.66606.421072.111133.03925.751007.22

IOzone

Record Size: 4Kb - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 8GB - Disk Test: Write PerformanceZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k140280420560700SE +/- 2.40, N = 3SE +/- 7.97, N = 3SE +/- 2.28, N = 3SE +/- 3.44, N = 3SE +/- 4.73, N = 3SE +/- 2.00, N = 3SE +/- 10.20, N = 3SE +/- 8.83, N = 15SE +/- 4.57, N = 3SE +/- 8.87, N = 15SE +/- 7.62, N = 15SE +/- 1.76, N = 3SE +/- 10.50, N = 15439.51622.91565.58243.93648.44446.27615.63395.38396.63552.19542.85426.39485.37

IOzone

Record Size: 64Kb - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 8GB - Disk Test: Read PerformanceZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k2004006008001000SE +/- 11.87, N = 3SE +/- 11.05, N = 3SE +/- 6.93, N = 3SE +/- 4.64, N = 15SE +/- 12.19, N = 3SE +/- 12.53, N = 3SE +/- 9.96, N = 5SE +/- 6.94, N = 11SE +/- 6.68, N = 3SE +/- 9.09, N = 15SE +/- 5.12, N = 3SE +/- 7.14, N = 3SE +/- 10.06, N = 4807.93833.63655.99408.54875.13835.14826.00792.23644.41810.39806.79799.85786.52

IOzone

Record Size: 64Kb - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 8GB - Disk Test: Write PerformanceZFS 128k - QCOW2 64k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 64kZFS 16k - QCOW2 16k - NTFS 16kZFS 4k - QCOW2 4k - NTFS - 4kZFS 128k - QCOW2 128k - NTFS 64kZFS 128k - QCOW2 128k - NTFS 4kZFS 128k - QCOW2 64k - NTFS 64kZFS 64k - QCOW2 64K - NTFS 4kZFS 16k - QCOW2 16k - NTFS 4kZFS 64k - QCOW2 64k - NTFS 16kZFS 64k - QCOW2 16k - NTFS 16kZFS 64k - QCOW2 4k - NTFS 4kZFS 64k - QCOW2 16k - NTFS 4k30060090012001500SE +/- 20.93, N = 12SE +/- 13.91, N = 15SE +/- 11.06, N = 15SE +/- 3.55, N = 4SE +/- 18.61, N = 4SE +/- 11.60, N = 6SE +/- 11.98, N = 3SE +/- 6.36, N = 3SE +/- 3.74, N = 3SE +/- 13.45, N = 5SE +/- 14.52, N = 5SE +/- 23.22, N = 13SE +/- 29.69, N = 15947.011154.15743.78266.341293.46951.011318.79808.15583.251125.911089.60908.61977.73


Phoronix Test Suite v10.8.4