2020-06-24-2159

no tuning, dedup=on, compression=on, sync=on, ashift=9

Compare your own system(s) to this result file with the Phoronix Test Suite by running the command: phoronix-test-suite benchmark 2006259-NI-2006251NI96
Jump To Table - Results

View

Do Not Show Noisy Results
Do Not Show Results With Incomplete Data
Do Not Show Results With Little Change/Spread
List Notable Results
Show Result Confidence Charts

Statistics

Show Overall Harmonic Mean(s)
Show Overall Geometric Mean
Show Wins / Losses Counts (Pie Chart)
Normalize Results
Remove Outliers Before Calculating Averages

Graph Settings

Force Line Graphs Where Applicable
Convert To Scalar Where Applicable
Prefer Vertical Bar Graphs

Multi-Way Comparison

Condense Multi-Option Tests Into Single Result Graphs

Table

Show Detailed System Result Table

Run Management

Highlight
Result
Hide
Result
Result
Identifier
Performance Per
Dollar
Date
Run
  Test
  Duration
zfs_6x_ssd_in_2x_raidz1
June 24 2020
  2 Hours, 24 Minutes
zfs_6x_ssd_in_2x_raidz1_ashift=12
June 24 2020
 
6 x 481GB HSCP0480S5xnNMRI
June 24 2020
  2 Hours, 29 Minutes
zfs_ashift=9_compression=on_dedup=off
June 25 2020
  1 Hour, 57 Minutes
zfs_ashift=9_compression=on_dedup=on
June 25 2020
  1 Hour, 19 Minutes
zfs_ashift=9_compression=on_dedup=on_sync=yes
June 25 2020
  1 Hour, 50 Minutes
Invert Hiding All Results Option
  1 Hour, 40 Minutes

Only show results where is faster than
Only show results matching title/arguments (delimit multiple options with a comma):
Do not show results matching title/arguments (delimit multiple options with a comma):

zfs_6x_ssd_in_2x_raidz1

Compiler Notes: --build=x86_64-redhat-linux --disable-libmpx --disable-libunwind-exceptions --enable-__cxa_atexit --enable-bootstrap --enable-cet --enable-checking=release --enable-gnu-indirect-function --enable-gnu-unique-object --enable-initfini-array --enable-languages=c,c++,fortran,lto --enable-multilib --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --mandir=/usr/share/man --with-arch_32=x86-64 --with-gcc-major-version-only --with-isl --with-linker-hash-style=gnu --with-tune=generic --without-cuda-driver
Disk Notes: MQ-DEADLINE / attr2,inode64,noquota,relatime,rw,seclabel
Processor Notes: Scaling Governor: intel_pstate performance - CPU Microcode: 0x43
Security Notes: SELinux + itlb_multihit: KVM: Mitigation of Split huge pages + l1tf: Mitigation of PTE Inversion; VMX: conditional cache flushes SMT vulnerable + mds: Mitigation of Clear buffers; SMT vulnerable + meltdown: Mitigation of PTI + spec_store_bypass: Mitigation of SSB disabled via prctl and seccomp + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Full generic retpoline IBPB: conditional IBRS_FW STIBP: conditional RSB filling + srbds: Not affected + tsx_async_abort: Not affected

zfs_6x_ssd_in_2x_raidz1_ashift=12

6 x 481GB HSCP0480S5xnNMRI

zfs_ashift=9_compression=on_dedup=off

zfs_ashift=9_compression=on_dedup=on

zfs_ashift=9_compression=on_dedup=on_sync=yes

Processor: 2 x Intel Xeon E5-2640 v3 @ 3.40GHz (16 Cores / 32 Threads), Motherboard: Dell 0CNCJW (2.11.0 BIOS), Chipset: Intel Xeon E7 v3/Xeon, Memory: 16 x GB DDR4-2133MT/s HMA42GR7MFR4N-TF, Disk: 6 x 481GB HSCP0480S5xnNMRI + 99GB PERC H730 Mini + 16GB Cruzer Blade + 16GB Cruzer Fit, Graphics: Matrox G200eR2, Monitor: DELL E196FP, Network: 2 x Intel 10-Gigabit X540-AT2 + 2 x Intel I350 + Mellanox MT27500

OS: CentOS Linux 8, Kernel: 4.18.0-193.6.3.el8_2.x86_64 (x86_64), Compiler: GCC 8.3.1 20191121, File-System: xfs, Screen Resolution: 1280x1024

2020-06-24-2159OpenBenchmarking.orgPhoronix Test Suite2 x Intel Xeon E5-2640 v3 @ 3.40GHz (16 Cores / 32 Threads)Dell 0CNCJW (2.11.0 BIOS)Intel Xeon E7 v3/Xeon16 x GB DDR4-2133MT/s HMA42GR7MFR4N-TF6 x 481GB HSCP0480S5xnNMRI + 99GB PERC H730 Mini + 16GB Cruzer Blade + 16GB Cruzer FitMatrox G200eR2DELL E196FP2 x Intel 10-Gigabit X540-AT2 + 2 x Intel I350 + Mellanox MT27500CentOS Linux 84.18.0-193.6.3.el8_2.x86_64 (x86_64)GCC 8.3.1 20191121xfs1280x1024ProcessorMotherboardChipsetMemoryDiskGraphicsMonitorNetworkOSKernelCompilerFile-SystemScreen Resolution2020-06-24-2159 BenchmarksSystem Logs- --build=x86_64-redhat-linux --disable-libmpx --disable-libunwind-exceptions --enable-__cxa_atexit --enable-bootstrap --enable-cet --enable-checking=release --enable-gnu-indirect-function --enable-gnu-unique-object --enable-initfini-array --enable-languages=c,c++,fortran,lto --enable-multilib --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --mandir=/usr/share/man --with-arch_32=x86-64 --with-gcc-major-version-only --with-isl --with-linker-hash-style=gnu --with-tune=generic --without-cuda-driver - MQ-DEADLINE / attr2,inode64,noquota,relatime,rw,seclabel- Scaling Governor: intel_pstate performance - CPU Microcode: 0x43- SELinux + itlb_multihit: KVM: Mitigation of Split huge pages + l1tf: Mitigation of PTE Inversion; VMX: conditional cache flushes SMT vulnerable + mds: Mitigation of Clear buffers; SMT vulnerable + meltdown: Mitigation of PTI + spec_store_bypass: Mitigation of SSB disabled via prctl and seccomp + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Full generic retpoline IBPB: conditional IBRS_FW STIBP: conditional RSB filling + srbds: Not affected + tsx_async_abort: Not affected

2020-06-24-2159fio: Rand Write - Sync - No - Yes - 4KB - /mnt/ztest1fio: Rand Write - Sync - No - Yes - 4KB - /mnt/ztest1fio: Rand Write - Sync - No - Yes - 16KB - /mnt/ztest1fio: Rand Write - Sync - No - Yes - 16KB - /mnt/ztest1fio: Rand Write - Sync - No - Yes - 64KB - /mnt/ztest1fio: Rand Write - Sync - No - Yes - 64KB - /mnt/ztest1fio: Seq Write - Linux AIO - No - Yes - 64KB - /mnt/ztest1fio: Seq Write - Linux AIO - No - Yes - 64KB - /mnt/ztest1fio: Seq Write - Sync - No - Yes - 16KB - /mnt/ztest1fio: Seq Write - Sync - No - Yes - 16KB - /mnt/ztest1fio: Seq Read - Sync - No - Yes - 64KB - /mnt/ztest1fio: Seq Read - Sync - No - Yes - 64KB - /mnt/ztest1fio: Rand Read - Linux AIO - No - Yes - 64KB - /mnt/ztest1fio: Rand Read - Linux AIO - No - Yes - 64KB - /mnt/ztest1fio: Rand Read - Linux AIO - No - Yes - 16KB - /mnt/ztest1fio: Rand Read - Linux AIO - No - Yes - 16KB - /mnt/ztest1fio: Seq Read - Sync - No - Yes - 16KB - /mnt/ztest1fio: Seq Read - Sync - No - Yes - 16KB - /mnt/ztest1fio: Seq Read - Linux AIO - No - Yes - 64KB - /mnt/ztest1fio: Seq Read - Linux AIO - No - Yes - 64KB - /mnt/ztest1fio: Rand Read - Sync - No - Yes - 4KB - /mnt/ztest1fio: Rand Read - Sync - No - Yes - 4KB - /mnt/ztest1fio: Seq Write - Sync - No - Yes - 4KB - /mnt/ztest1fio: Seq Read - Linux AIO - No - Yes - 16KB - /mnt/ztest1fio: Seq Write - Sync - No - Yes - 4KB - /mnt/ztest1fio: Seq Read - Linux AIO - No - Yes - 16KB - /mnt/ztest1fio: Seq Read - Sync - No - Yes - 4KB - /mnt/ztest1fio: Seq Read - Sync - No - Yes - 4KB - /mnt/ztest1fio: Seq Write - Sync - No - Yes - 64KB - /mnt/ztest1fio: Seq Write - Sync - No - Yes - 64KB - /mnt/ztest1fio: Seq Read - Linux AIO - No - Yes - 4KB - /mnt/ztest1fio: Seq Read - Linux AIO - No - Yes - 4KB - /mnt/ztest1fio: Rand Write - Linux AIO - No - Yes - 64KB - /mnt/ztest1fio: Rand Write - Linux AIO - No - Yes - 64KB - /mnt/ztest1fio: Rand Read - Linux AIO - No - Yes - 4KB - /mnt/ztest1fio: Rand Read - Linux AIO - No - Yes - 4KB - /mnt/ztest1fio: Seq Write - Linux AIO - No - Yes - 16KB - /mnt/ztest1fio: Seq Write - Linux AIO - No - Yes - 16KB - /mnt/ztest1fio: Seq Write - Linux AIO - No - Yes - 4KB - /mnt/ztest1fio: Seq Write - Linux AIO - No - Yes - 4KB - /mnt/ztest1fio: Rand Write - Linux AIO - No - Yes - 16KB - /mnt/ztest1fio: Rand Write - Linux AIO - No - Yes - 16KB - /mnt/ztest1fio: Rand Write - Linux AIO - No - Yes - 4KB - /mnt/ztest1fio: Rand Write - Linux AIO - No - Yes - 4KB - /mnt/ztest1fio: Rand Read - Sync - No - Yes - 64KB - /mnt/ztest1fio: Rand Read - Sync - No - Yes - 64KB - /mnt/ztest1fio: Rand Read - Sync - No - Yes - 16KB - /mnt/ztest1fio: Rand Read - Sync - No - Yes - 16KB - /mnt/ztest1zfs_6x_ssd_in_2x_raidz1zfs_6x_ssd_in_2x_raidz1_ashift=126 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes211540274566771818793006718112896716251040007716748226545040921466672288325420833343857016724213394670017600017933327531396357667334752106108627833316112570722333387214239106715733361542225662541332127277345531622672535231588874952077618652986720723316715811011507833348976625641531460002283324420766744957190023746792870318066718033328201416362667335672097111728633316132580022000086014419223316100062846364726556672186816742601543332407321823676153396116692670019613160018621190007663348096586741171486672321323320700044547126024766796874318060019033328211447371000348002176113128966715442470022233386814419218715700061439640620463001817030043941606672511344880336633310372009321332150343671862119000786004911678834242151000235832952110004534725332473339667541826671926672858143536766735300220811212870001608257332266678861471941331553336064710073655033215717004503163000254733384900640331005197431725215534467183211700085792536272767454816066725073535226067476576233257000100275118933319200029561484379667353332209113629100016072570022966789714809470015533360846633729545002137996049981826672860OpenBenchmarking.org

Flexible IO Tester

Fio is an advanced disk benchmark that depends upon the kernel's AIO access library. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Random Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1zfs_6x_ssd_in_2x_raidz16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=on_sync=yeszfs_ashift=9_compression=on_dedup=on70140210280350SE +/- 3.16, N = 15SE +/- 3.26, N = 15SE +/- 3.21, N = 3SE +/- 2.33, N = 32112313213333441. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Random Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1zfs_6x_ssd_in_2x_raidz16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=on_sync=yeszfs_ashift=9_compression=on_dedup=on20K40K60K80K100KSE +/- 803.41, N = 15SE +/- 840.57, N = 15SE +/- 1097.47, N = 3SE +/- 1096.97, N = 3SE +/- 638.57, N = 354027588878236784900880331. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Random Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1zfs_6x_ssd_in_2x_raidz16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=on_sync=yeszfs_ashift=9_compression=on_dedup=on14K28K42K56K70KSE +/- 717.25, N = 3SE +/- 473.11, N = 15SE +/- 88.19, N = 3SE +/- 145.30, N = 3SE +/- 328.30, N = 345667495206153364033663331. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Random Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1zfs_6x_ssd_in_2x_raidz16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=on_sync=yeszfs_ashift=9_compression=on_dedup=on2004006008001000SE +/- 10.11, N = 3SE +/- 7.20, N = 15SE +/- 1.45, N = 3SE +/- 5.13, N = 3718776961100510371. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Random Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1zfs_ashift=9_compression=on_dedup=off6 x 481GB HSCP0480S5xnNMRIzfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=on_sync=yeszfs_ashift=9_compression=on_dedup=on400800120016002000SE +/- 20.83, N = 4SE +/- 32.20, N = 3SE +/- 25.71, N = 3SE +/- 21.79, N = 3SE +/- 15.88, N = 3166918651879197420091. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Random Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1zfs_ashift=9_compression=on_dedup=off6 x 481GB HSCP0480S5xnNMRIzfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=on_sync=yeszfs_ashift=9_compression=on_dedup=on7K14K21K28K35KSE +/- 331.66, N = 4SE +/- 504.42, N = 3SE +/- 433.33, N = 3SE +/- 292.62, N = 4SE +/- 260.34, N = 326700298673006731725321331. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Sequential Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1zfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=off6 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes5001000150020002500SE +/- 21.62, N = 3SE +/- 30.83, N = 3SE +/- 33.95, N = 3SE +/- 5.51, N = 3SE +/- 4.41, N = 3181119612072215021551. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Sequential Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1zfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=off6 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes7K14K21K28K35KSE +/- 352.77, N = 3SE +/- 422.30, N = 4SE +/- 523.87, N = 3SE +/- 88.19, N = 3SE +/- 66.67, N = 328967316003316734367344671. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Sequential Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest16 x 481GB HSCP0480S5xnNMRIzfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=on_sync=yeszfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=on400800120016002000SE +/- 17.99, N = 6SE +/- 7.64, N = 3SE +/- 18.48, N = 3SE +/- 7.23, N = 3SE +/- 16.68, N = 3158116251832186218621. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Sequential Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest16 x 481GB HSCP0480S5xnNMRIzfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=on_sync=yeszfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=on30K60K90K120K150KSE +/- 1152.89, N = 6SE +/- 577.35, N = 3SE +/- 1154.70, N = 3SE +/- 577.35, N = 3SE +/- 1000.00, N = 31011501040001170001190001190001. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Sequential Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1zfs_ashift=9_compression=on_dedup=offzfs_6x_ssd_in_2x_raidz16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes20K40K60K80K100KSE +/- 328.30, N = 3SE +/- 66.67, N = 3SE +/- 176.38, N = 3SE +/- 100.00, N = 3SE +/- 702.86, N = 1276633771677833378600857921. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Sequential Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1zfs_ashift=9_compression=on_dedup=offzfs_6x_ssd_in_2x_raidz16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes11002200330044005500SE +/- 4.73, N = 3SE +/- 4.04, N = 3SE +/- 10.17, N = 3SE +/- 6.23, N = 3SE +/- 43.97, N = 12480948224897491153621. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Random Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1zfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=off6 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes16K32K48K64K80KSE +/- 28.87, N = 4SE +/- 425.57, N = 3SE +/- 583.81, N = 16SE +/- 793.06, N = 6SE +/- 185.59, N = 365450658676625667883727671. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Random Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1zfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=off6 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes10002000300040005000SE +/- 3.53, N = 3SE +/- 28.04, N = 3SE +/- 37.97, N = 15SE +/- 49.30, N = 6SE +/- 12.66, N = 3409241174153424245481. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Random Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest16 x 481GB HSCP0480S5xnNMRIzfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes30K60K90K120K150KSE +/- 333.33, N = 3SE +/- 881.92, N = 3SE +/- 577.35, N = 3SE +/- 333.33, N = 31460001466671486671510001606671. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Random Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest16 x 481GB HSCP0480S5xnNMRIzfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes5001000150020002500SE +/- 1.86, N = 3SE +/- 5.17, N = 3SE +/- 16.37, N = 3SE +/- 9.61, N = 3SE +/- 3.84, N = 3228322882321235825071. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Sequential Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1zfs_ashift=9_compression=on_dedup=off6 x 481GB HSCP0480S5xnNMRIzfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes8001600240032004000SE +/- 3.28, N = 3SE +/- 25.32, N = 3SE +/- 3.76, N = 3SE +/- 7.22, N = 3SE +/- 35.37, N = 15323332443254329535351. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Sequential Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1zfs_ashift=9_compression=on_dedup=off6 x 481GB HSCP0480S5xnNMRIzfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes50K100K150K200K250KSE +/- 1855.92, N = 3SE +/- 333.33, N = 3SE +/- 577.35, N = 3SE +/- 2266.67, N = 152070002076672083332110002260671. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Sequential Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1zfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=off6 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes10002000300040005000SE +/- 4.37, N = 3SE +/- 37.20, N = 15SE +/- 9.17, N = 3SE +/- 6.12, N = 3SE +/- 14.93, N = 3438544544495453447651. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Sequential Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1zfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=off6 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes16K32K48K64K80KSE +/- 66.67, N = 3SE +/- 594.56, N = 15SE +/- 152.75, N = 3SE +/- 88.19, N = 3SE +/- 233.33, N = 370167712607190072533762331. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Random Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest16 x 481GB HSCP0480S5xnNMRIzfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=on_sync=yes60K120K180K240K300KSE +/- 2302.93, N = 15SE +/- 2946.94, N = 15SE +/- 1855.92, N = 3SE +/- 3179.80, N = 3SE +/- 577.35, N = 32374672421332473332476672570001. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Random Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest16 x 481GB HSCP0480S5xnNMRIzfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=on_sync=yes2004006008001000SE +/- 8.94, N = 15SE +/- 11.46, N = 15SE +/- 6.98, N = 3SE +/- 12.78, N = 3SE +/- 2.60, N = 392894696696810021. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Sequential Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1zfs_6x_ssd_in_2x_raidz16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=on_sync=yeszfs_ashift=9_compression=on_dedup=on160320480640800SE +/- 4.26, N = 3SE +/- 5.81, N = 3SE +/- 7.06, N = 3SE +/- 4.00, N = 3SE +/- 2.73, N = 37007037437517541. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Sequential Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1zfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=off6 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes40K80K120K160K200KSE +/- 577.35, N = 3SE +/- 2357.97, N = 5SE +/- 333.33, N = 3SE +/- 1666.67, N = 3SE +/- 333.33, N = 31760001806001806671826671893331. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Sequential Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1zfs_6x_ssd_in_2x_raidz16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=on_sync=yeszfs_ashift=9_compression=on_dedup=on40K80K120K160K200KSE +/- 881.92, N = 3SE +/- 1452.97, N = 3SE +/- 1763.83, N = 3SE +/- 1000.00, N = 3SE +/- 666.67, N = 31793331803331903331920001926671. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Sequential Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1zfs_6x_ssd_in_2x_raidz16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes6001200180024003000SE +/- 5.90, N = 3SE +/- 2.40, N = 3SE +/- 35.95, N = 5SE +/- 26.21, N = 3SE +/- 4.81, N = 3275328202821285829561. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Sequential Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1zfs_6x_ssd_in_2x_raidz16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=on_sync=yes30060090012001500SE +/- 4.18, N = 3SE +/- 2.40, N = 3SE +/- 7.69, N = 3SE +/- 21.61, N = 3SE +/- 4.73, N = 3139614161435144714841. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Sequential Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1zfs_6x_ssd_in_2x_raidz16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=on_sync=yes80K160K240K320K400KSE +/- 1333.33, N = 3SE +/- 666.67, N = 3SE +/- 1763.83, N = 3SE +/- 5567.76, N = 3SE +/- 1201.85, N = 33576673626673676673710003796671. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Sequential Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1zfs_6x_ssd_in_2x_raidz16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes8K16K24K32K40KSE +/- 319.83, N = 4SE +/- 166.67, N = 3SE +/- 503.32, N = 3SE +/- 57.74, N = 3SE +/- 202.76, N = 333475335673480035300353331. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Sequential Write - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest16 x 481GB HSCP0480S5xnNMRIzfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes5001000150020002500SE +/- 10.20, N = 3SE +/- 31.54, N = 4SE +/- 30.28, N = 3SE +/- 2.65, N = 3SE +/- 12.86, N = 3209721062176220822091. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Sequential Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1zfs_6x_ssd_in_2x_raidz16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=on_sync=yes2004006008001000SE +/- 14.90, N = 3SE +/- 2.73, N = 3SE +/- 3.84, N = 3SE +/- 6.35, N = 3SE +/- 1.86, N = 3108611171121113111361. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Sequential Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1zfs_6x_ssd_in_2x_raidz16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=on_sync=yes60K120K180K240K300KSE +/- 3844.19, N = 3SE +/- 666.67, N = 3SE +/- 1452.97, N = 32783332863332870002896672910001. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Random Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1zfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=on_sync=yeszfs_ashift=9_compression=on_dedup=onzfs_6x_ssd_in_2x_raidz16 x 481GB HSCP0480S5xnNMRI30060090012001500SE +/- 13.45, N = 3SE +/- 26.54, N = 3SE +/- 20.14, N = 15SE +/- 4.10, N = 3154416071608161116131. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Random Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest1zfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=on_sync=yeszfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=on6 x 481GB HSCP0480S5xnNMRI6K12K18K24K30KSE +/- 208.17, N = 3SE +/- 313.88, N = 15SE +/- 433.33, N = 3SE +/- 57.74, N = 324700257002570725733258001. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Random Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=offzfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes50K100K150K200K250KSE +/- 577.35, N = 3SE +/- 1666.67, N = 3SE +/- 3382.96, N = 3SE +/- 1201.85, N = 3SE +/- 333.33, N = 32200002223332233332266672296671. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Random Read - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=offzfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes2004006008001000SE +/- 2.65, N = 3SE +/- 5.84, N = 3SE +/- 13.38, N = 3SE +/- 4.06, N = 3SE +/- 2.08, N = 38608688728868971. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Sequential Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1zfs_6x_ssd_in_2x_raidz16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes30060090012001500SE +/- 16.33, N = 3SE +/- 19.47, N = 3SE +/- 12.05, N = 15SE +/- 8.54, N = 3SE +/- 1.53, N = 3142314411441147114801. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Sequential Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1zfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=off6 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes20K40K60K80K100KSE +/- 1052.51, N = 3SE +/- 772.68, N = 15SE +/- 1234.68, N = 3SE +/- 569.60, N = 3SE +/- 100.00, N = 391067921879223394133947001. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Sequential Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1zfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yeszfs_ashift=9_compression=on_dedup=offzfs_6x_ssd_in_2x_raidz16 x 481GB HSCP0480S5xnNMRI30K60K90K120K150KSE +/- 881.92, N = 3SE +/- 666.67, N = 3SE +/- 1763.83, N = 3SE +/- 1527.53, N = 31553331553331570001573331610001. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Sequential Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1zfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yeszfs_ashift=9_compression=on_dedup=offzfs_6x_ssd_in_2x_raidz16 x 481GB HSCP0480S5xnNMRI140280420560700SE +/- 2.89, N = 3SE +/- 2.33, N = 3SE +/- 1.00, N = 3SE +/- 6.89, N = 3SE +/- 5.90, N = 36066086146156281. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Random Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1zfs_ashift=9_compression=on_dedup=offzfs_6x_ssd_in_2x_raidz16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=on_sync=yeszfs_ashift=9_compression=on_dedup=on10K20K30K40K50KSE +/- 329.62, N = 15SE +/- 506.30, N = 8SE +/- 825.66, N = 14SE +/- 66.67, N = 3SE +/- 513.16, N = 339640422254636446633471001. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Random Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest1zfs_ashift=9_compression=on_dedup=offzfs_6x_ssd_in_2x_raidz16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=on_sync=yeszfs_ashift=9_compression=on_dedup=on160320480640800SE +/- 5.18, N = 15SE +/- 6.87, N = 8SE +/- 12.87, N = 14SE +/- 1.00, N = 3SE +/- 8.09, N = 36206627267297361. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Random Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1zfs_ashift=9_compression=on_dedup=offzfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=on_sync=yeszfs_ashift=9_compression=on_dedup=on6 x 481GB HSCP0480S5xnNMRI12K24K36K48K60KSE +/- 678.23, N = 4SE +/- 648.93, N = 3SE +/- 202.76, N = 3SE +/- 1279.40, N = 1246300541335450055033556671. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Random Write - Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: /mnt/ztest1zfs_ashift=9_compression=on_dedup=offzfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=on_sync=yeszfs_ashift=9_compression=on_dedup=on6 x 481GB HSCP0480S5xnNMRI50100150200250SE +/- 2.48, N = 4SE +/- 2.40, N = 3SE +/- 0.58, N = 3SE +/- 5.09, N = 121812122132152181. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Random Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=onzfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=on_sync=yes20K40K60K80K100KSE +/- 33.33, N = 3SE +/- 115.47, N = 3SE +/- 351.19, N = 3SE +/- 1917.72, N = 15SE +/- 1378.16, N = 1568167703007170072773799601. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Random Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 64KB - Disk Target: /mnt/ztest16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=offzfs_ashift=9_compression=on_dedup=onzfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=on_sync=yes11002200330044005500SE +/- 2.40, N = 3SE +/- 8.67, N = 3SE +/- 2.52, N = 3SE +/- 121.70, N = 15SE +/- 86.11, N = 15426043944503455349981. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.18Type: Random Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=offzfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes40K80K120K160K200KSE +/- 666.67, N = 3SE +/- 666.67, N = 3SE +/- 3412.12, N = 15SE +/- 577.35, N = 3SE +/- 666.67, N = 31543331606671622671630001826671. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.18Type: Random Read - Engine: Sync - Buffered: No - Direct: Yes - Block Size: 16KB - Disk Target: /mnt/ztest16 x 481GB HSCP0480S5xnNMRIzfs_ashift=9_compression=on_dedup=offzfs_6x_ssd_in_2x_raidz1zfs_ashift=9_compression=on_dedup=onzfs_ashift=9_compression=on_dedup=on_sync=yes6001200180024003000SE +/- 9.68, N = 3SE +/- 8.33, N = 3SE +/- 53.47, N = 15SE +/- 8.82, N = 3SE +/- 10.00, N = 3240725112535254728601. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native -lrt -laio -lpthread -lm -ldl

48 Results Shown

Flexible IO Tester:
  Rand Write - Sync - No - Yes - 4KB - /mnt/ztest1:
    MB/s
    IOPS
  Rand Write - Sync - No - Yes - 16KB - /mnt/ztest1:
    IOPS
    MB/s
  Rand Write - Sync - No - Yes - 64KB - /mnt/ztest1:
    MB/s
    IOPS
  Seq Write - Linux AIO - No - Yes - 64KB - /mnt/ztest1:
    MB/s
    IOPS
  Seq Write - Sync - No - Yes - 16KB - /mnt/ztest1:
    MB/s
    IOPS
  Seq Read - Sync - No - Yes - 64KB - /mnt/ztest1:
    IOPS
    MB/s
  Rand Read - Linux AIO - No - Yes - 64KB - /mnt/ztest1:
    IOPS
    MB/s
  Rand Read - Linux AIO - No - Yes - 16KB - /mnt/ztest1:
    IOPS
    MB/s
  Seq Read - Sync - No - Yes - 16KB - /mnt/ztest1:
    MB/s
    IOPS
  Seq Read - Linux AIO - No - Yes - 64KB - /mnt/ztest1:
    MB/s
    IOPS
  Rand Read - Sync - No - Yes - 4KB - /mnt/ztest1:
    IOPS
    MB/s
  Seq Write - Sync - No - Yes - 4KB - /mnt/ztest1:
    MB/s
  Seq Read - Linux AIO - No - Yes - 16KB - /mnt/ztest1:
    IOPS
  Seq Write - Sync - No - Yes - 4KB - /mnt/ztest1:
    IOPS
  Seq Read - Linux AIO - No - Yes - 16KB - /mnt/ztest1:
    MB/s
  Seq Read - Sync - No - Yes - 4KB - /mnt/ztest1:
    MB/s
    IOPS
  Seq Write - Sync - No - Yes - 64KB - /mnt/ztest1:
    IOPS
    MB/s
  Seq Read - Linux AIO - No - Yes - 4KB - /mnt/ztest1:
    MB/s
    IOPS
  Rand Write - Linux AIO - No - Yes - 64KB - /mnt/ztest1:
    MB/s
    IOPS
  Rand Read - Linux AIO - No - Yes - 4KB - /mnt/ztest1:
    IOPS
    MB/s
  Seq Write - Linux AIO - No - Yes - 16KB - /mnt/ztest1:
    MB/s
    IOPS
  Seq Write - Linux AIO - No - Yes - 4KB - /mnt/ztest1:
    IOPS
    MB/s
  Rand Write - Linux AIO - No - Yes - 16KB - /mnt/ztest1:
    IOPS
    MB/s
  Rand Write - Linux AIO - No - Yes - 4KB - /mnt/ztest1:
    IOPS
    MB/s
  Rand Read - Sync - No - Yes - 64KB - /mnt/ztest1:
    IOPS
    MB/s
  Rand Read - Sync - No - Yes - 16KB - /mnt/ztest1:
    IOPS
    MB/s

zfs_6x_ssd_in_2x_raidz1

Compiler Notes: --build=x86_64-redhat-linux --disable-libmpx --disable-libunwind-exceptions --enable-__cxa_atexit --enable-bootstrap --enable-cet --enable-checking=release --enable-gnu-indirect-function --enable-gnu-unique-object --enable-initfini-array --enable-languages=c,c++,fortran,lto --enable-multilib --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --mandir=/usr/share/man --with-arch_32=x86-64 --with-gcc-major-version-only --with-isl --with-linker-hash-style=gnu --with-tune=generic --without-cuda-driver
Disk Notes: MQ-DEADLINE / attr2,inode64,noquota,relatime,rw,seclabel
Processor Notes: Scaling Governor: intel_pstate performance - CPU Microcode: 0x43
Security Notes: SELinux + itlb_multihit: KVM: Mitigation of Split huge pages + l1tf: Mitigation of PTE Inversion; VMX: conditional cache flushes SMT vulnerable + mds: Mitigation of Clear buffers; SMT vulnerable + meltdown: Mitigation of PTI + spec_store_bypass: Mitigation of SSB disabled via prctl and seccomp + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Full generic retpoline IBPB: conditional IBRS_FW STIBP: conditional RSB filling + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 24 June 2020 21:59 by user .

zfs_6x_ssd_in_2x_raidz1_ashift=12

Compiler Notes: --build=x86_64-redhat-linux --disable-libmpx --disable-libunwind-exceptions --enable-__cxa_atexit --enable-bootstrap --enable-cet --enable-checking=release --enable-gnu-indirect-function --enable-gnu-unique-object --enable-initfini-array --enable-languages=c,c++,fortran,lto --enable-multilib --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --mandir=/usr/share/man --with-arch_32=x86-64 --with-gcc-major-version-only --with-isl --with-linker-hash-style=gnu --with-tune=generic --without-cuda-driver
Disk Notes: MQ-DEADLINE / attr2,inode64,noquota,relatime,rw,seclabel
Processor Notes: Scaling Governor: intel_pstate performance - CPU Microcode: 0x43
Security Notes: SELinux + itlb_multihit: KVM: Mitigation of Split huge pages + l1tf: Mitigation of PTE Inversion; VMX: conditional cache flushes SMT vulnerable + mds: Mitigation of Clear buffers; SMT vulnerable + meltdown: Mitigation of PTI + spec_store_bypass: Mitigation of SSB disabled via prctl and seccomp + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Full generic retpoline IBPB: conditional IBRS_FW STIBP: conditional RSB filling + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 24 June 2020 23:34 by user .

6 x 481GB HSCP0480S5xnNMRI

Compiler Notes: --build=x86_64-redhat-linux --disable-libmpx --disable-libunwind-exceptions --enable-__cxa_atexit --enable-bootstrap --enable-cet --enable-checking=release --enable-gnu-indirect-function --enable-gnu-unique-object --enable-initfini-array --enable-languages=c,c++,fortran,lto --enable-multilib --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --mandir=/usr/share/man --with-arch_32=x86-64 --with-gcc-major-version-only --with-isl --with-linker-hash-style=gnu --with-tune=generic --without-cuda-driver
Disk Notes: MQ-DEADLINE / attr2,inode64,noquota,relatime,rw,seclabel
Processor Notes: Scaling Governor: intel_pstate performance - CPU Microcode: 0x43
Security Notes: SELinux + itlb_multihit: KVM: Mitigation of Split huge pages + l1tf: Mitigation of PTE Inversion; VMX: conditional cache flushes SMT vulnerable + mds: Mitigation of Clear buffers; SMT vulnerable + meltdown: Mitigation of PTI + spec_store_bypass: Mitigation of SSB disabled via prctl and seccomp + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Full generic retpoline IBPB: conditional IBRS_FW STIBP: conditional RSB filling + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 24 June 2020 23:35 by user .

zfs_ashift=9_compression=on_dedup=off

Compiler Notes: --build=x86_64-redhat-linux --disable-libmpx --disable-libunwind-exceptions --enable-__cxa_atexit --enable-bootstrap --enable-cet --enable-checking=release --enable-gnu-indirect-function --enable-gnu-unique-object --enable-initfini-array --enable-languages=c,c++,fortran,lto --enable-multilib --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --mandir=/usr/share/man --with-arch_32=x86-64 --with-gcc-major-version-only --with-isl --with-linker-hash-style=gnu --with-tune=generic --without-cuda-driver
Disk Notes: MQ-DEADLINE / attr2,inode64,noquota,relatime,rw,seclabel
Processor Notes: Scaling Governor: intel_pstate performance - CPU Microcode: 0x43
Security Notes: SELinux + itlb_multihit: KVM: Mitigation of Split huge pages + l1tf: Mitigation of PTE Inversion; VMX: conditional cache flushes SMT vulnerable + mds: Mitigation of Clear buffers; SMT vulnerable + meltdown: Mitigation of PTI + spec_store_bypass: Mitigation of SSB disabled via prctl and seccomp + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Full generic retpoline IBPB: conditional IBRS_FW STIBP: conditional RSB filling + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 25 June 2020 11:25 by user .

zfs_ashift=9_compression=on_dedup=on

Compiler Notes: --build=x86_64-redhat-linux --disable-libmpx --disable-libunwind-exceptions --enable-__cxa_atexit --enable-bootstrap --enable-cet --enable-checking=release --enable-gnu-indirect-function --enable-gnu-unique-object --enable-initfini-array --enable-languages=c,c++,fortran,lto --enable-multilib --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --mandir=/usr/share/man --with-arch_32=x86-64 --with-gcc-major-version-only --with-isl --with-linker-hash-style=gnu --with-tune=generic --without-cuda-driver
Disk Notes: MQ-DEADLINE / attr2,inode64,noquota,relatime,rw,seclabel
Processor Notes: Scaling Governor: intel_pstate performance - CPU Microcode: 0x43
Security Notes: SELinux + itlb_multihit: KVM: Mitigation of Split huge pages + l1tf: Mitigation of PTE Inversion; VMX: conditional cache flushes SMT vulnerable + mds: Mitigation of Clear buffers; SMT vulnerable + meltdown: Mitigation of PTI + spec_store_bypass: Mitigation of SSB disabled via prctl and seccomp + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Full generic retpoline IBPB: conditional IBRS_FW STIBP: conditional RSB filling + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 25 June 2020 12:43 by user .

zfs_ashift=9_compression=on_dedup=on_sync=yes

Processor: 2 x Intel Xeon E5-2640 v3 @ 3.40GHz (16 Cores / 32 Threads), Motherboard: Dell 0CNCJW (2.11.0 BIOS), Chipset: Intel Xeon E7 v3/Xeon, Memory: 16 x GB DDR4-2133MT/s HMA42GR7MFR4N-TF, Disk: 6 x 481GB HSCP0480S5xnNMRI + 99GB PERC H730 Mini + 16GB Cruzer Blade + 16GB Cruzer Fit, Graphics: Matrox G200eR2, Monitor: DELL E196FP, Network: 2 x Intel 10-Gigabit X540-AT2 + 2 x Intel I350 + Mellanox MT27500

OS: CentOS Linux 8, Kernel: 4.18.0-193.6.3.el8_2.x86_64 (x86_64), Compiler: GCC 8.3.1 20191121, File-System: xfs, Screen Resolution: 1280x1024

Compiler Notes: --build=x86_64-redhat-linux --disable-libmpx --disable-libunwind-exceptions --enable-__cxa_atexit --enable-bootstrap --enable-cet --enable-checking=release --enable-gnu-indirect-function --enable-gnu-unique-object --enable-initfini-array --enable-languages=c,c++,fortran,lto --enable-multilib --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --mandir=/usr/share/man --with-arch_32=x86-64 --with-gcc-major-version-only --with-isl --with-linker-hash-style=gnu --with-tune=generic --without-cuda-driver
Disk Notes: MQ-DEADLINE / attr2,inode64,noquota,relatime,rw,seclabel
Processor Notes: Scaling Governor: intel_pstate performance - CPU Microcode: 0x43
Security Notes: SELinux + itlb_multihit: KVM: Mitigation of Split huge pages + l1tf: Mitigation of PTE Inversion; VMX: conditional cache flushes SMT vulnerable + mds: Mitigation of Clear buffers; SMT vulnerable + meltdown: Mitigation of PTI + spec_store_bypass: Mitigation of SSB disabled via prctl and seccomp + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Full generic retpoline IBPB: conditional IBRS_FW STIBP: conditional RSB filling + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 25 June 2020 14:32 by user .