Ubuntu 19.10 ZFS vs. EXT4

AMD Ryzen 9 3900X ZFS vs. EXT4 tests by Michael Larabel for a future article.

Compare your own system(s) to this result file with the Phoronix Test Suite by running the command: phoronix-test-suite benchmark 1910109-AS-UBUNTUZFS73
Jump To Table - Results

View

Do Not Show Noisy Results
Do Not Show Results With Incomplete Data
Do Not Show Results With Little Change/Spread
List Notable Results

Limit displaying results to tests within:

CPU Massive 2 Tests
Database Test Suite 3 Tests
Disk Test Suite 3 Tests
Common Kernel Benchmarks 2 Tests
Multi-Core 2 Tests
Programmer / Developer System Benchmarks 2 Tests
Server 3 Tests

Statistics

Show Overall Harmonic Mean(s)
Show Overall Geometric Mean
Show Geometric Means Per-Suite/Category
Show Wins / Losses Counts (Pie Chart)
Normalize Results
Remove Outliers Before Calculating Averages

Graph Settings

Force Line Graphs Where Applicable
Convert To Scalar Where Applicable
Prefer Vertical Bar Graphs

Multi-Way Comparison

Condense Multi-Option Tests Into Single Result Graphs

Table

Show Detailed System Result Table

Run Management

Highlight
Result
Hide
Result
Result
Identifier
Performance Per
Dollar
Date
Run
  Test
  Duration
ZFS
October 10 2019
  1 Hour, 7 Minutes
EXT4
October 10 2019
  1 Hour, 22 Minutes
Invert Hiding All Results Option
  1 Hour, 15 Minutes
Only show results matching title/arguments (delimit multiple options with a comma):
Do not show results matching title/arguments (delimit multiple options with a comma):


Ubuntu 19.10 ZFS vs. EXT4ProcessorMotherboardChipsetMemoryDiskGraphicsAudioMonitorNetworkOSKernelDesktopDisplay ServerDisplay DriverOpenGLCompilerFile-SystemScreen ResolutionZFSEXT4AMD Ryzen 9 3900X 12-Core @ 3.80GHz (12 Cores / 24 Threads)ASUS ROG CROSSHAIR VIII HERO (WI-FI) (1001 BIOS)AMD Starship/Matisse16384MB2000GB Force MP600AMD Radeon RX 56/64 8GB (1630/945MHz)AMD Vega 10 HDMI AudioASUS VP28URealtek Device 8125 + Intel I211 + Intel Device 2723Ubuntu 19.105.3.0-17-generic (x86_64)GNOME Shell 3.34.1X Server 1.20.5amdgpu 19.0.14.5 Mesa 19.2.0 (LLVM 9.0.0)GCC 9.2.1 20191008zfs3840x2160ext4OpenBenchmarking.orgCompiler Details- --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2 --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-offload-targets=nvptx-none,hsa --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details- Scaling Governor: acpi-cpufreq ondemandDisk Scheduler Details- ZFS: NONEJava Details- OpenJDK Runtime Environment (build 11.0.5-ea+9-post-Ubuntu-1ubuntu1)Python Details- Python 2.7.16+ + Python 3.7.5rc1Security Details- l1tf: Not affected + mds: Not affected + meltdown: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl and seccomp + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Full AMD retpoline IBPB: conditional STIBP: always-on RSB filling Disk Details- EXT4: NONE / errors=remount-ro,relatime,rw

ZFS vs. EXT4 ComparisonPhoronix Test SuiteBaseline+203.3%+203.3%+406.6%+406.6%+609.9%+609.9%813.1%812.3%425.5%425.4%242.4%196.8%156.3%84.9%70%65.3%62%14.7%5.1%Rand Write - Linux AIO - No - No - 4KBRand Write - Linux AIO - No - No - 4KBRand Read - Linux AIO - No - No - 4KBRand Read - Linux AIO - No - No - 4KBInitial CreateRead Compiled Tree1.F.1.S161.7%Timed Time - Size 1,000Seq FillRand FillRead While WritingCompileSeq Read - Linux AIO - No - No - 2MB37.6%WritesTime To CompileFlexible IO TesterFlexible IO TesterFlexible IO TesterFlexible IO TesterCompile BenchCompile BenchFS-MarkSQLite SpeedtestFacebook RocksDBFacebook RocksDBFacebook RocksDBCompile BenchFlexible IO TesterApache CassandraTimed GCC CompilationZFSEXT4

Ubuntu 19.10 ZFS vs. EXT4fio: Rand Read - Linux AIO - No - No - 4KB - Default Test Directoryfio: Rand Read - Linux AIO - No - No - 4KB - Default Test Directoryfio: Rand Write - Linux AIO - No - No - 4KB - Default Test Directoryfio: Rand Write - Linux AIO - No - No - 4KB - Default Test Directoryfio: Seq Read - Linux AIO - No - No - 2MB - Default Test Directoryfs-mark: 1000 Files, 1MB Sizecompilebench: Compilecompilebench: Initial Createcompilebench: Read Compiled Treebuild-gcc: Time To Compilesqlite-speedtest: Timed Time - Size 1,000cassandra: Writesrocksdb: Rand Fillrocksdb: Rand Readrocksdb: Seq Fillrocksdb: Read While WritingZFSEXT428372433102.402623340861513.081505.70209.281343.29705.54145.999567456439563237260600927172349414873806679352393332970578.202438.50716.663987.50671.1656.961096919594286339626111108422848469OpenBenchmarking.org

Flexible IO Tester

Fio is an advanced disk benchmark that depends upon the kernel's AIO access library. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.16Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: No - Block Size: 4KB - Disk Target: Default Test DirectoryZFSEXT430060090012001500SE +/- 2.33, N = 3SE +/- 8.35, N = 328314871. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -march=native -ll -lrt -laio -lm -lpthread -ldl
OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.16Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: No - Block Size: 4KB - Disk Target: Default Test DirectoryZFSEXT430060090012001500Min: 281 / Avg: 283.33 / Max: 288Min: 1470 / Avg: 1486.67 / Max: 14961. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -march=native -ll -lrt -laio -lm -lpthread -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.16Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: No - Block Size: 4KB - Disk Target: Default Test DirectoryZFSEXT480K160K240K320K400KSE +/- 633.33, N = 3SE +/- 2333.33, N = 3724333806671. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -march=native -ll -lrt -laio -lm -lpthread -ldl
OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.16Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: No - Block Size: 4KB - Disk Target: Default Test DirectoryZFSEXT470K140K210K280K350KMin: 71800 / Avg: 72433.33 / Max: 73700Min: 376000 / Avg: 380666.67 / Max: 3830001. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -march=native -ll -lrt -laio -lm -lpthread -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.16Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: No - Block Size: 4KB - Disk Target: Default Test DirectoryZFSEXT42004006008001000SE +/- 1.60, N = 3SE +/- 52.81, N = 12102.40935.001. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -march=native -ll -lrt -laio -lm -lpthread -ldl
OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.16Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: No - Block Size: 4KB - Disk Target: Default Test DirectoryZFSEXT4160320480640800Min: 99.2 / Avg: 102.4 / Max: 104Min: 611 / Avg: 934.83 / Max: 12331. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -march=native -ll -lrt -laio -lm -lpthread -ldl

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.16Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: No - Block Size: 4KB - Disk Target: Default Test DirectoryZFSEXT450K100K150K200K250KSE +/- 417.67, N = 3SE +/- 13569.87, N = 12262332393331. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -march=native -ll -lrt -laio -lm -lpthread -ldl
OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.16Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: No - Block Size: 4KB - Disk Target: Default Test DirectoryZFSEXT440K80K120K160K200KMin: 25400 / Avg: 26233.33 / Max: 26700Min: 156000 / Avg: 239333.33 / Max: 3160001. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -march=native -ll -lrt -laio -lm -lpthread -ldl

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.16Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: No - Block Size: 2MB - Disk Target: Default Test DirectoryZFSEXT49001800270036004500SE +/- 2.19, N = 3SE +/- 307.39, N = 12408629701. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -march=native -ll -lrt -laio -lm -lpthread -ldl
OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.16Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: No - Block Size: 2MB - Disk Target: Default Test DirectoryZFSEXT47001400210028003500Min: 4082 / Avg: 4086.33 / Max: 4089Min: 2158 / Avg: 2969.5 / Max: 47741. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -march=native -ll -lrt -laio -lm -lpthread -ldl

FS-Mark

FS_Mark is designed to test a system's file-system performance. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 1000 Files, 1MB SizeZFSEXT430060090012001500SE +/- 14.58, N = 9SE +/- 2.69, N = 31513.08578.201. (CC) gcc options: -static
OpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 1000 Files, 1MB SizeZFSEXT430060090012001500Min: 1467.3 / Avg: 1513.08 / Max: 1577.2Min: 573.7 / Avg: 578.2 / Max: 5831. (CC) gcc options: -static

Compile Bench

Compilebench tries to age a filesystem by simulating some of the disk IO common in creating, compiling, patching, stating and reading kernel trees. It indirectly measures how well filesystems can maintain directory locality as the disk fills up and directories age. This current test is setup to use the makej mode with 10 initial directories Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: CompileZFSEXT45001000150020002500SE +/- 19.76, N = 3SE +/- 41.19, N = 31505.702438.50
OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: CompileZFSEXT4400800120016002000Min: 1466.4 / Avg: 1505.7 / Max: 1528.97Min: 2397.31 / Avg: 2438.5 / Max: 2520.87

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Initial CreateZFSEXT4150300450600750SE +/- 2.22, N = 3SE +/- 3.45, N = 3209.28716.66
OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Initial CreateZFSEXT4130260390520650Min: 205.1 / Avg: 209.28 / Max: 212.67Min: 711.33 / Avg: 716.66 / Max: 723.11

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Read Compiled TreeZFSEXT49001800270036004500SE +/- 9.17, N = 3SE +/- 61.38, N = 31343.293987.50
OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Read Compiled TreeZFSEXT47001400210028003500Min: 1328.48 / Avg: 1343.29 / Max: 1360.05Min: 3926.12 / Avg: 3987.5 / Max: 4110.25

Timed GCC Compilation

This test times how long it takes to build the GNU Compiler Collection (GCC). Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterTimed GCC Compilation 8.2Time To CompileZFSEXT4150300450600750SE +/- 1.39, N = 3SE +/- 0.60, N = 3705.54671.16
OpenBenchmarking.orgSeconds, Fewer Is BetterTimed GCC Compilation 8.2Time To CompileZFSEXT4120240360480600Min: 703.38 / Avg: 705.54 / Max: 708.14Min: 669.97 / Avg: 671.16 / Max: 671.77

SQLite Speedtest

This is a benchmark of SQLite's speedtest1 benchmark program with an increased problem size of 1,000. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgSeconds, Fewer Is BetterSQLite Speedtest 3.30Timed Time - Size 1,000ZFSEXT4306090120150SE +/- 0.45, N = 3SE +/- 0.24, N = 3145.9956.961. (CC) gcc options: -O2 -ldl -lpthread
OpenBenchmarking.orgSeconds, Fewer Is BetterSQLite Speedtest 3.30Timed Time - Size 1,000ZFSEXT4306090120150Min: 145.18 / Avg: 145.99 / Max: 146.74Min: 56.49 / Avg: 56.96 / Max: 57.261. (CC) gcc options: -O2 -ldl -lpthread

Apache Cassandra

This is a benchmark of the Apache Cassandra NoSQL database management system making use of cassandra-stress. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgOp/s, More Is BetterApache Cassandra 3.11.4Test: WritesZFSEXT420K40K60K80K100KSE +/- 1313.71, N = 3SE +/- 1079.62, N = 895674109691
OpenBenchmarking.orgOp/s, More Is BetterApache Cassandra 3.11.4Test: WritesZFSEXT420K40K60K80K100KMin: 93068 / Avg: 95674 / Max: 97267Min: 102362 / Avg: 109690.5 / Max: 111802

Facebook RocksDB

This is a benchmark of Facebook's RocksDB as an embeddable persistent key-value store for fast storage based on Google's LevelDB. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgOp/s, More Is BetterFacebook RocksDB 6.3.6Test: Random FillZFSEXT4200K400K600K800K1000KSE +/- 1525.01, N = 3SE +/- 6909.48, N = 35643959594281. (CXX) g++ options: -O3 -march=native -std=c++11 -fno-builtin-memcmp -fno-rtti -rdynamic -lpthread
OpenBenchmarking.orgOp/s, More Is BetterFacebook RocksDB 6.3.6Test: Random FillZFSEXT4170K340K510K680K850KMin: 561647 / Avg: 564395 / Max: 566915Min: 946346 / Avg: 959428 / Max: 9698251. (CXX) g++ options: -O3 -march=native -std=c++11 -fno-builtin-memcmp -fno-rtti -rdynamic -lpthread

OpenBenchmarking.orgOp/s, More Is BetterFacebook RocksDB 6.3.6Test: Random ReadZFSEXT414M28M42M56M70MSE +/- 948389.35, N = 3SE +/- 666074.26, N = 363237260633962611. (CXX) g++ options: -O3 -march=native -std=c++11 -fno-builtin-memcmp -fno-rtti -rdynamic -lpthread
OpenBenchmarking.orgOp/s, More Is BetterFacebook RocksDB 6.3.6Test: Random ReadZFSEXT411M22M33M44M55MMin: 61342693 / Avg: 63237260.33 / Max: 64263841Min: 62065219 / Avg: 63396260.67 / Max: 641088011. (CXX) g++ options: -O3 -march=native -std=c++11 -fno-builtin-memcmp -fno-rtti -rdynamic -lpthread

OpenBenchmarking.orgOp/s, More Is BetterFacebook RocksDB 6.3.6Test: Sequential FillZFSEXT4200K400K600K800K1000KSE +/- 4171.10, N = 3SE +/- 6209.26, N = 360092711108421. (CXX) g++ options: -O3 -march=native -std=c++11 -fno-builtin-memcmp -fno-rtti -rdynamic -lpthread
OpenBenchmarking.orgOp/s, More Is BetterFacebook RocksDB 6.3.6Test: Sequential FillZFSEXT4200K400K600K800K1000KMin: 593066 / Avg: 600926.67 / Max: 607276Min: 1101684 / Avg: 1110842.33 / Max: 11226851. (CXX) g++ options: -O3 -march=native -std=c++11 -fno-builtin-memcmp -fno-rtti -rdynamic -lpthread

OpenBenchmarking.orgOp/s, More Is BetterFacebook RocksDB 6.3.6Test: Read While WritingZFSEXT4600K1200K1800K2400K3000KSE +/- 26538.96, N = 3SE +/- 34441.37, N = 5172349428484691. (CXX) g++ options: -O3 -march=native -std=c++11 -fno-builtin-memcmp -fno-rtti -rdynamic -lpthread
OpenBenchmarking.orgOp/s, More Is BetterFacebook RocksDB 6.3.6Test: Read While WritingZFSEXT4500K1000K1500K2000K2500KMin: 1670416 / Avg: 1723493.67 / Max: 1750177Min: 2761117 / Avg: 2848468.8 / Max: 29710511. (CXX) g++ options: -O3 -march=native -std=c++11 -fno-builtin-memcmp -fno-rtti -rdynamic -lpthread

16 Results Shown

Flexible IO Tester:
  Rand Read - Linux AIO - No - No - 4KB - Default Test Directory:
    MB/s
    IOPS
  Rand Write - Linux AIO - No - No - 4KB - Default Test Directory:
    MB/s
    IOPS
  Seq Read - Linux AIO - No - No - 2MB - Default Test Directory:
    MB/s
FS-Mark
Compile Bench:
  Compile
  Initial Create
  Read Compiled Tree
Timed GCC Compilation
SQLite Speedtest
Apache Cassandra
Facebook RocksDB:
  Rand Fill
  Rand Read
  Seq Fill
  Read While Writing