Btrfs RAID Benchmarks on Linux 5.14

Linux 5.14 RAID NVMe SSD benchmarks on WD_BLACK SN850 dual drives. Benchmarks by Michael Larabel.

HTML result view exported from: https://openbenchmarking.org/result/2108260-PTS-STORAGET40&sor.

Btrfs RAID Benchmarks on Linux 5.14ProcessorMotherboardChipsetMemoryDiskGraphicsAudioMonitorNetworkOSKernelDesktopDisplay ServerOpenGLVulkanCompilerFile-SystemScreen ResolutionSingleRAID0RAID1AMD Ryzen 9 5900X 12-Core @ 3.70GHz (12 Cores / 24 Threads)ASUS ROG CROSSHAIR VIII HERO (3501 BIOS)AMD Starship/Matisse16GB1000GB Western Digital WD_BLACK SN850 1TB + 1000GB Western Digital WDS100T1X0E-00AFY0AMD DIMGREY_CAVEFISH 8GB (2900/1000MHz)AMD Device ab28ASUS VP28URealtek RTL8125 2.5GbE + Intel I211Ubuntu 20.045.14.0-051400rc7-generic (x86_64) 20210822GNOME Shell 3.36.9X Server 1.20.114.6 Mesa 21.3.0-devel (git-2b4b310 2021-08-21 focal-oibaf-ppa) (LLVM 12.0.1)1.2.185GCC 9.3.0btrfs3840x21602 x 1000GB Western Digital WDS100T1X0E-00AFY0 + 1000GB Western Digital WD_BLACK SN850 1TBOpenBenchmarking.orgKernel Details- Transparent Huge Pages: madviseCompiler Details- --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2 --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-9-HskZEa/gcc-9-9.3.0/debian/tmp-nvptx/usr,hsa --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Disk Details- Single: NONE / relatime,rw,space_cache,ssd,subvol=/,subvolid=5 / Block Size: 4096- RAID0: NONE / relatime,rw,space_cache,ssd,subvol=/,subvolid=5 / RAID0 Block Size: 4096- RAID1: NONE / relatime,rw,space_cache,ssd,subvol=/,subvolid=5 / RAID1 Block Size: 4096Processor Details- Scaling Governor: acpi-cpufreq ondemand (Boost: Enabled) - CPU Microcode: 0xa201009Security Details- itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl and seccomp + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Full AMD retpoline IBPB: conditional IBRS_FW STIBP: always-on RSB filling + srbds: Not affected + tsx_async_abort: Not affected

Btrfs RAID Benchmarks on Linux 5.14sqlite: 8fio: Rand Read - IO_uring - No - Yes - 4KB - Default Test Directoryfio: Rand Write - IO_uring - No - Yes - 4KB - Default Test Directoryfio: Seq Read - IO_uring - No - Yes - 2MB - Default Test Directoryfio: Seq Write - IO_uring - No - Yes - 2MB - Default Test Directoryfs-mark: 1000 Files, 1MB Sizefs-mark: 5000 Files, 1MB Size, 4 Threadsfs-mark: 4000 Files, 32 Sub Dirs, 1MB Sizedbench: 6ior: 16MB - Default Test Directorymysqlslap: 128mysqlslap: 256mysqlslap: 512pgbench: 100 - 50 - Read Writepgbench: 100 - 50 - Read Write - Average Latencypgbench: 1000 - 50 - Read Onlypgbench: 1000 - 50 - Read Only - Average Latencypgbench: 1000 - 50 - Read Writepgbench: 1000 - 50 - Read Write - Average LatencySingleRAID0RAID124.34555433314333332533273872.21841.0859.22005.652291.05856725706243562.0533113790.161203612.45624.069553000158667331864691085.92617.61065.92042.412909.75860701687244302.0473123610.160205392.43525.79053533313966733253238947.91912.7905.21982.632454.13854710684236212.123100290.161194462.572OpenBenchmarking.org

SQLite

Threads / Copies: 8

OpenBenchmarking.orgSeconds, Fewer Is BetterSQLite 3.30.1Threads / Copies: 8RAID0SingleRAID1612182430SE +/- 0.22, N = 7SE +/- 0.31, N = 15SE +/- 0.15, N = 324.0724.3525.791. (CC) gcc options: -O2 -lz -lm -ldl -lpthread

Flexible IO Tester

Type: Random Read - Engine: IO_uring - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.25Type: Random Read - Engine: IO_uring - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectorySingleRAID0RAID1120K240K360K480K600KSE +/- 2905.93, N = 3SE +/- 881.92, N = 35543335530005353331. (CC) gcc options: -rdynamic -ll -lnuma -lrt -lz -lpthread -lm -ldl -laio -lcurl -lssl -lcrypto -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

Flexible IO Tester

Type: Random Write - Engine: IO_uring - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.25Type: Random Write - Engine: IO_uring - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryRAID0SingleRAID130K60K90K120K150KSE +/- 881.92, N = 3SE +/- 1201.85, N = 3SE +/- 1763.83, N = 31586671433331396671. (CC) gcc options: -rdynamic -ll -lnuma -lrt -lz -lpthread -lm -ldl -laio -lcurl -lssl -lcrypto -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

Flexible IO Tester

Type: Sequential Read - Engine: IO_uring - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.25Type: Sequential Read - Engine: IO_uring - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryRAID1RAID0Single7001400210028003500SE +/- 21.52, N = 3SE +/- 12.44, N = 3SE +/- 39.61, N = 33325331832531. (CC) gcc options: -rdynamic -ll -lnuma -lrt -lz -lpthread -lm -ldl -laio -lcurl -lssl -lcrypto -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

Flexible IO Tester

Type: Sequential Write - Engine: IO_uring - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.25Type: Sequential Write - Engine: IO_uring - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryRAID0SingleRAID114002800420056007000SE +/- 3.71, N = 3SE +/- 0.88, N = 3SE +/- 1.15, N = 36469327332381. (CC) gcc options: -rdynamic -ll -lnuma -lrt -lz -lpthread -lm -ldl -laio -lcurl -lssl -lcrypto -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

FS-Mark

Test: 1000 Files, 1MB Size

OpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 1000 Files, 1MB SizeRAID0RAID1Single2004006008001000SE +/- 1.54, N = 3SE +/- 8.33, N = 3SE +/- 3.88, N = 31085.9947.9872.21. (CC) gcc options: -static

FS-Mark

Test: 5000 Files, 1MB Size, 4 Threads

OpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 5000 Files, 1MB Size, 4 ThreadsRAID0RAID1Single6001200180024003000SE +/- 5.57, N = 3SE +/- 4.42, N = 3SE +/- 21.56, N = 32617.61912.71841.01. (CC) gcc options: -static

FS-Mark

Test: 4000 Files, 32 Sub Dirs, 1MB Size

OpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 4000 Files, 32 Sub Dirs, 1MB SizeRAID0RAID1Single2004006008001000SE +/- 9.58, N = 3SE +/- 7.27, N = 3SE +/- 2.65, N = 31065.9905.2859.21. (CC) gcc options: -static

Dbench

Client Count: 6

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 6RAID0SingleRAID1400800120016002000SE +/- 1.79, N = 3SE +/- 3.25, N = 3SE +/- 3.77, N = 32042.412005.651982.631. (CC) gcc options: -lpopt -O2

IOR

Block Size: 16MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterIOR 3.3.0Block Size: 16MB - Disk Target: Default Test DirectoryRAID0RAID1Single6001200180024003000SE +/- 9.12, N = 3SE +/- 26.32, N = 3SE +/- 10.10, N = 32909.752454.132291.05MIN: 2289.94 / MAX: 3517.65MIN: 2101.18 / MAX: 2865.91MIN: 1995.52 / MAX: 2735.911. (CC) gcc options: -O2 -lm -pthread -lmpi

MariaDB

Clients: 128

OpenBenchmarking.orgQueries Per Second, More Is BetterMariaDB 10.6.4Clients: 128RAID0SingleRAID12004006008001000SE +/- 11.17, N = 3SE +/- 2.40, N = 3SE +/- 1.90, N = 38608568541. (CXX) g++ options: -pie -fPIC -fstack-protector -O3 -pthread -lbz2 -lnuma -lpcre2-8 -lcrypt -laio -lz -lm -lssl -lcrypto -lpthread -ldl

MariaDB

Clients: 256

OpenBenchmarking.orgQueries Per Second, More Is BetterMariaDB 10.6.4Clients: 256SingleRAID1RAID0160320480640800SE +/- 3.04, N = 3SE +/- 2.60, N = 3SE +/- 8.54, N = 37257107011. (CXX) g++ options: -pie -fPIC -fstack-protector -O3 -pthread -lbz2 -lnuma -lpcre2-8 -lcrypt -laio -lz -lm -lssl -lcrypto -lpthread -ldl

MariaDB

Clients: 512

OpenBenchmarking.orgQueries Per Second, More Is BetterMariaDB 10.6.4Clients: 512SingleRAID0RAID1150300450600750SE +/- 1.20, N = 3SE +/- 3.39, N = 3SE +/- 1.43, N = 37066876841. (CXX) g++ options: -pie -fPIC -fstack-protector -O3 -pthread -lbz2 -lnuma -lpcre2-8 -lcrypt -laio -lz -lm -lssl -lcrypto -lpthread -ldl

PostgreSQL pgbench

Scaling Factor: 100 - Clients: 50 - Mode: Read Write

OpenBenchmarking.orgTPS, More Is BetterPostgreSQL pgbench 13.0Scaling Factor: 100 - Clients: 50 - Mode: Read WriteRAID0SingleRAID15K10K15K20K25KSE +/- 155.64, N = 3SE +/- 104.55, N = 3SE +/- 97.06, N = 32443024356236211. (CC) gcc options: -fno-strict-aliasing -fwrapv -O2 -lpgcommon -lpgport -lpq -lpthread -lrt -ldl -lm

PostgreSQL pgbench

Scaling Factor: 100 - Clients: 50 - Mode: Read Write - Average Latency

OpenBenchmarking.orgms, Fewer Is BetterPostgreSQL pgbench 13.0Scaling Factor: 100 - Clients: 50 - Mode: Read Write - Average LatencyRAID0SingleRAID10.4770.9541.4311.9082.385SE +/- 0.013, N = 3SE +/- 0.009, N = 3SE +/- 0.009, N = 32.0472.0532.1201. (CC) gcc options: -fno-strict-aliasing -fwrapv -O2 -lpgcommon -lpgport -lpq -lpthread -lrt -ldl -lm

PostgreSQL pgbench

Scaling Factor: 1000 - Clients: 50 - Mode: Read Only

OpenBenchmarking.orgTPS, More Is BetterPostgreSQL pgbench 13.0Scaling Factor: 1000 - Clients: 50 - Mode: Read OnlyRAID0SingleRAID170K140K210K280K350KSE +/- 3920.14, N = 3SE +/- 2881.18, N = 3SE +/- 3356.29, N = 33123613113793100291. (CC) gcc options: -fno-strict-aliasing -fwrapv -O2 -lpgcommon -lpgport -lpq -lpthread -lrt -ldl -lm

PostgreSQL pgbench

Scaling Factor: 1000 - Clients: 50 - Mode: Read Only - Average Latency

OpenBenchmarking.orgms, Fewer Is BetterPostgreSQL pgbench 13.0Scaling Factor: 1000 - Clients: 50 - Mode: Read Only - Average LatencyRAID0SingleRAID10.03620.07240.10860.14480.181SE +/- 0.002, N = 3SE +/- 0.002, N = 3SE +/- 0.002, N = 30.1600.1610.1611. (CC) gcc options: -fno-strict-aliasing -fwrapv -O2 -lpgcommon -lpgport -lpq -lpthread -lrt -ldl -lm

PostgreSQL pgbench

Scaling Factor: 1000 - Clients: 50 - Mode: Read Write

OpenBenchmarking.orgTPS, More Is BetterPostgreSQL pgbench 13.0Scaling Factor: 1000 - Clients: 50 - Mode: Read WriteRAID0SingleRAID14K8K12K16K20KSE +/- 148.43, N = 3SE +/- 101.55, N = 3SE +/- 124.29, N = 32053920361194461. (CC) gcc options: -fno-strict-aliasing -fwrapv -O2 -lpgcommon -lpgport -lpq -lpthread -lrt -ldl -lm

PostgreSQL pgbench

Scaling Factor: 1000 - Clients: 50 - Mode: Read Write - Average Latency

OpenBenchmarking.orgms, Fewer Is BetterPostgreSQL pgbench 13.0Scaling Factor: 1000 - Clients: 50 - Mode: Read Write - Average LatencyRAID0SingleRAID10.57871.15741.73612.31482.8935SE +/- 0.018, N = 3SE +/- 0.012, N = 3SE +/- 0.016, N = 32.4352.4562.5721. (CC) gcc options: -fno-strict-aliasing -fwrapv -O2 -lpgcommon -lpgport -lpq -lpthread -lrt -ldl -lm

Phoronix Test Suite v10.8.4