Bcachefs Linux File-System Testing + Benchmarks

Bcachefs Linux file-system benchmarking against Btrfs, EXT4, F2FS, and XFS on an HDD and SSD. Benchmarks by Michael Larabel for a future article.

HTML result view exported from: https://openbenchmarking.org/result/1805304-AR-BCACHEFST07&grs&rdt.

ProcessorMotherboardChipsetMemoryDiskGraphicsMonitorNetworkOSKernelDesktopCompilerFile-SystemScreen ResolutionSSDHDD Bcachefs EXT4 Btrfs F2FS XFS XFS Btrfs EXT4 Bcachefs2 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads)TYAN S7106 (V1.00 BIOS)Intel Sky Lake-E DMI3 Registers12 x 8192 MB DDR4-2666MT/s Micron 9ASF1G72PZ-2G6B1256GB Samsung SSD 850 + 2000GB Seagate ST2000DM006-2DM1 + 2 x 120GB TOSHIBA-TR150ASPEED ASPEED FamilyVE228Intel I210 Gigabit ConnectionUbuntu 18.044.16.0-bcachefs (x86_64)GNOME Shell 3.28.1GCC 7.3.0BcacheFS1920x1080ext4btrfsf2fsxfsbtrfsext4BcacheFSOpenBenchmarking.orgCompiler Details- --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-as=/usr/bin/x86_64-linux-gnu-as --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-ld=/usr/bin/x86_64-linux-gnu-ld --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic --without-cuda-driver -v Disk Details- SSD: Bcachefs: CFQ / relatime,rw- SSD: EXT4: CFQ / data=ordered,relatime,rw- SSD: Btrfs: CFQ / relatime,rw,space_cache,ssd,subvol=/,subvolid=5- SSD: F2FS: CFQ / acl,active_logs=6,background_gc=on,extent_cache,flush_merge,inline_data,inline_dentry,inline_xattr,lazytime,mode=adaptive,no_heap,relatime,rw,user_xattr - SSD: XFS: CFQ / attr2,inode64,noquota,relatime,rw- HDD: XFS: CFQ / attr2,inode64,noquota,relatime,rw- HDD: Btrfs: CFQ / relatime,rw,space_cache,subvol=/,subvolid=5- HDD: EXT4: CFQ / data=ordered,relatime,rw- HDD: Bcachefs: CFQ / relatime,rwProcessor Details- Scaling Governor: intel_pstate powersavePython Details- Python 2.7.15rc1 + Python 3.6.5Security Details- KPTI + __user pointer sanitization + Full generic retpoline Protection

compilebench: Initial Createsqlite: Timed SQLite Insertionsdbench: 6fs-mark: 4000 Files, 32 Sub Dirs, 1MB Sizefio: Rand Read - Linux AIO - No - Yes - 4KB - Default Test Directoryfio: Seq Read - Linux AIO - No - Yes - 4KB - Default Test Directoryfs-mark: 5000 Files, 1MB Size, 4 Threadsstartup-time: Only Seq Reads - GNOME Terminalpgbench: Buffer Test - Normal Load - Read Writetiobench: Rand Write - 128MB - 16tiobench: Write - 128MB - 16iozone: 4Kb - 8GB - Write Performancefs-mark: 1000 Files, 1MB Sizefio: Seq Write - Linux AIO - No - Yes - 4KB - Default Test Directoryfio: Rand Write - Linux AIO - No - Yes - 4KB - Default Test Directorystartup-time: Seq Reads + Writes - GNOME TerminalSSDHDD Bcachefs EXT4 Btrfs F2FS XFS XFS Btrfs EXT4 Bcachefs506.9758.45313.2144.3533.5719750.935.36905.24268.189.3237.9048.6347.8050.4349.88519.9640.98358.5341.2822521150.675.282293.36404.9010.46103.9357.9741528054.20122.6887.64243.1744.7021023150.705.341386.0094.115.8592.3877.0082.3580.1554.79559.4034.30277.6344.5521322551.205.381600.03126.457.3788.9557.4241128050.45410.1136.98438.7343.4021120252.305.302485.86396.168.92105.6996.3737627147.57419.41407.9317.9619.851.5618136.675.31304.16162.0819.831771.2050.00288.901010.4242.8221.231.531.4937.875.27578.88166.0823.174.1628.2349.66515.36685.2018.9617.301.5516245.475.36313.95135.1017.631551.0752.52238.34707.3730.1221.301.391.6134.575.29625.33136.3122.1781.0785.5348.12OpenBenchmarking.org

Flexible IO Tester

Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

SSDHDDOpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryBcachefsEXT4BtrfsF2FSXFS15K30K45K60K75KSE +/- 208.17, N = 6SE +/- 176.38, N = 3SE +/- 678.07, N = 6SE +/- 208.17, N = 3SE +/- 983.76, N = 3SE +/- 6350.00, N = 2SE +/- 290.59, N = 3129007183320567717006943316550219671. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrt -laio -lz -lm -lpthread -ldl

Compile Bench

Test: Initial Create

SSDHDDOpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Initial CreateBcachefsEXT4BtrfsF2FSXFS120240360480600SE +/- 0.67, N = 3SE +/- 2.81, N = 3SE +/- 2.42, N = 6SE +/- 2.19, N = 3SE +/- 5.64, N = 5SE +/- 4.91, N = 3SE +/- 8.61, N = 6SE +/- 4.00, N = 3SE +/- 6.04, N = 6506.97519.96122.68559.40410.11419.41288.90515.36238.34

SQLite

Timed SQLite Insertions

SSDHDDOpenBenchmarking.orgSeconds, Fewer Is BetterSQLite 3.22Timed SQLite InsertionsBcachefsEXT4BtrfsF2FSXFS2004006008001000SE +/- 0.78, N = 3SE +/- 0.86, N = 6SE +/- 0.96, N = 3SE +/- 0.32, N = 3SE +/- 0.61, N = 4SE +/- 4.56, N = 3SE +/- 4.34, N = 3SE +/- 5.39, N = 3SE +/- 3.47, N = 358.4540.9887.6434.3036.98407.931010.42685.20707.371. (CC) gcc options: -O2 -lz -ldl -lpthread

Dbench

Client Count: 6

SSDHDDOpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 6BcachefsEXT4BtrfsF2FSXFS100200300400500SE +/- 6.09, N = 3SE +/- 10.98, N = 6SE +/- 3.95, N = 3SE +/- 0.35, N = 3SE +/- 2.86, N = 3SE +/- 0.00, N = 3SE +/- 0.63, N = 3SE +/- 0.31, N = 4SE +/- 0.33, N = 3313.21358.53243.17277.63438.7317.9642.8218.9630.121. (CC) gcc options: -lpopt -O2

FS-Mark

Test: 4000 Files, 32 Sub Dirs, 1MB Size

SSDHDDOpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 4000 Files, 32 Sub Dirs, 1MB SizeBcachefsEXT4BtrfsF2FSXFS1020304050SE +/- 1.43, N = 6SE +/- 0.60, N = 6SE +/- 1.02, N = 6SE +/- 1.26, N = 6SE +/- 0.85, N = 6SE +/- 0.34, N = 6SE +/- 0.23, N = 3SE +/- 0.00, N = 3SE +/- 0.10, N = 344.3541.2844.7044.5543.4019.8521.2317.3021.301. (CC) gcc options: -static

Flexible IO Tester

Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

SSDHDDOpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryBcachefsEXT4BtrfsF2FSXFS50100150200250SE +/- 0.03, N = 3SE +/- 0.67, N = 3SE +/- 0.01, N = 3SE +/- 0.01, N = 3SE +/- 0.01, N = 3SE +/- 0.01, N = 333.57225.00210.00213.00211.001.561.531.551.391. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrt -laio -lz -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

SSDHDDOpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryBcachefsEXT4BtrfsF2FSXFS50100150200250SE +/- 1.76, N = 3SE +/- 4.95, N = 6SE +/- 7.53, N = 6SE +/- 0.33, N = 3SE +/- 0.67, N = 3SE +/- 0.01, N = 3SE +/- 0.01, N = 3197.00211.00231.00225.00202.00181.001.49162.001.611. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrt -laio -lz -lm -lpthread -ldl

FS-Mark

Test: 5000 Files, 1MB Size, 4 Threads

SSDHDDOpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 5000 Files, 1MB Size, 4 ThreadsBcachefsEXT4BtrfsF2FSXFS1224364860SE +/- 0.27, N = 3SE +/- 0.27, N = 3SE +/- 0.60, N = 3SE +/- 0.40, N = 3SE +/- 0.10, N = 3SE +/- 0.19, N = 3SE +/- 0.12, N = 3SE +/- 0.35, N = 3SE +/- 0.20, N = 350.9350.6750.7051.2052.3036.6737.8745.4734.571. (CC) gcc options: -static

Flexible IO Tester

Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

SSDHDDOpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryBcachefsEXT4BtrfsF2FSXFS13K26K39K52K65KSE +/- 504.42, N = 3SE +/- 1267.81, N = 6SE +/- 1939.89, N = 6SE +/- 88.19, N = 3SE +/- 120.19, N = 3SE +/- 57.74, N = 3SE +/- 33.33, N = 3503335400059150576335156746300414331. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrt -laio -lz -lm -lpthread -ldl

Flexible IO Tester

Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryEXT4BtrfsF2FSXFS12K24K36K48K60KSE +/- 33.33, N = 3SE +/- 260.34, N = 3SE +/- 33.33, N = 3SE +/- 351.19, N = 3576675346754533537001. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrt -laio -lz -lm -lpthread -ldl

Application Start-up Time

Background I/O Mix: Only Sequential Reads - Application To Start: GNOME Terminal

SSDHDDOpenBenchmarking.orgsec, Fewer Is BetterApplication Start-up Time 2.3.0Background I/O Mix: Only Sequential Reads - Application To Start: GNOME TerminalBcachefsEXT4BtrfsF2FSXFS1.21052.4213.63154.8426.0525SE +/- 0.02, N = 3SE +/- 0.05, N = 3SE +/- 0.01, N = 3SE +/- 0.04, N = 3SE +/- 0.07, N = 3SE +/- 0.04, N = 3SE +/- 0.06, N = 3SE +/- 0.04, N = 3SE +/- 0.04, N = 35.365.285.345.385.305.315.275.365.29

PostgreSQL pgbench

Scaling: Buffer Test - Test: Normal Load - Mode: Read Write

SSDHDDOpenBenchmarking.orgTPS, More Is BetterPostgreSQL pgbench 10.3Scaling: Buffer Test - Test: Normal Load - Mode: Read WriteBcachefsEXT4BtrfsF2FSXFS5001000150020002500SE +/- 112.07, N = 6SE +/- 29.84, N = 3SE +/- 87.14, N = 6SE +/- 72.84, N = 6SE +/- 116.18, N = 6SE +/- 7.05, N = 6SE +/- 10.48, N = 6SE +/- 5.78, N = 6SE +/- 5.38, N = 3905.242293.361386.001600.032485.86304.16578.88313.95625.331. (CC) gcc options: -fno-strict-aliasing -fwrapv -O2 -lpgcommon -lpgport -lpq -lpthread -lrt -lcrypt -ldl -lm

Threaded I/O Tester

Test: Random Write - Size Per Thread: 128MB - Thread Count: 16

OpenBenchmarking.orgMB/s, More Is BetterThreaded I/O Tester 20170503Test: Random Write - Size Per Thread: 128MB - Thread Count: 16BcachefsEXT4BtrfsF2FSXFS90180270360450SE +/- 69.81, N = 6SE +/- 26.90, N = 6SE +/- 8.70, N = 6SE +/- 13.96, N = 6SE +/- 11.66, N = 6268.18404.9094.11126.45396.161. (CC) gcc options: -O2

Threaded I/O Tester

Test: Write - Size Per Thread: 128MB - Thread Count: 16

OpenBenchmarking.orgMB/s, More Is BetterThreaded I/O Tester 20170503Test: Write - Size Per Thread: 128MB - Thread Count: 16BcachefsEXT4BtrfsF2FSXFS3691215SE +/- 0.28, N = 6SE +/- 0.16, N = 5SE +/- 0.04, N = 3SE +/- 0.06, N = 3SE +/- 0.15, N = 39.3210.465.857.378.921. (CC) gcc options: -O2

IOzone

Record Size: 4Kb - File Size: 8GB - Disk Test: Write Performance

SSDHDDOpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 8GB - Disk Test: Write PerformanceBcachefsEXT4BtrfsF2FSXFS4080120160200SE +/- 1.89, N = 6SE +/- 2.31, N = 6SE +/- 3.54, N = 6SE +/- 2.53, N = 6SE +/- 1.74, N = 3SE +/- 7.35, N = 6SE +/- 4.97, N = 6SE +/- 0.30, N = 3SE +/- 2.45, N = 637.90103.9392.3888.95105.69162.08166.08135.10136.311. (CC) gcc options: -O3

FS-Mark

Test: 1000 Files, 1MB Size

SSDHDDOpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 1000 Files, 1MB SizeBcachefsEXT4BtrfsF2FSXFS20406080100SE +/- 4.68, N = 6SE +/- 11.04, N = 6SE +/- 10.26, N = 6SE +/- 10.47, N = 6SE +/- 4.75, N = 6SE +/- 0.03, N = 3SE +/- 0.26, N = 3SE +/- 0.03, N = 3SE +/- 0.07, N = 348.6357.9777.0057.4296.3719.8323.1717.6322.171. (CC) gcc options: -static

Flexible IO Tester

Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

SSDHDDOpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryBcachefsEXT4BtrfsF2FSXFS20K40K60K80K100KSE +/- 458.26, N = 3SE +/- 333.33, N = 3SE +/- 697.61, N = 6SE +/- 6462.22, N = 6SE +/- 759.80, N = 4SE +/- 272.85, N = 3SE +/- 260.34, N = 31190010666721100105000957834517539767204331. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrt -laio -lz -lm -lpthread -ldl

Flexible IO Tester

Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

SSDHDDOpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryBcachefsEXT4BtrfsF2FSXFS90180270360450SE +/- 0.58, N = 3SE +/- 1.20, N = 3SE +/- 2.68, N = 6SE +/- 24.98, N = 6SE +/- 2.93, N = 4SE +/- 0.38, N = 6SE +/- 0.88, N = 3SE +/- 0.74, N = 347.80415.0082.35411.00376.00177.004.16155.0081.071. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrt -laio -lz -lm -lpthread -ldl

Flexible IO Tester

Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

SSDHDDOpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryBcachefsEXT4BtrfsF2FSXFS60120180240300SE +/- 0.83, N = 6SE +/- 0.67, N = 3SE +/- 2.57, N = 6SE +/- 3.76, N = 3SE +/- 0.03, N = 6SE +/- 13.15, N = 6SE +/- 0.06, N = 6SE +/- 0.81, N = 350.43280.0080.15280.00271.001.2028.231.0785.531. (CC) gcc options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrt -laio -lz -lm -lpthread -ldl

Application Start-up Time

Background I/O Mix: Sequential Reads + Writes - Application To Start: GNOME Terminal

SSDHDDOpenBenchmarking.orgsec, Fewer Is BetterApplication Start-up Time 2.3.0Background I/O Mix: Sequential Reads + Writes - Application To Start: GNOME TerminalBcachefsEXT4BtrfsF2FSXFS1224364860SE +/- 2.45, N = 6SE +/- 2.28, N = 6SE +/- 2.04, N = 6SE +/- 3.16, N = 6SE +/- 0.38, N = 3SE +/- 1.49, N = 6SE +/- 1.08, N = 6SE +/- 1.98, N = 6SE +/- 1.52, N = 649.8854.2054.7950.4547.5750.0049.6652.5248.12

Phoronix Test Suite v10.8.4