Btrfs RAID Benchmarks on Linux 5.14

Linux 5.14 RAID NVMe SSD benchmarks on WD_BLACK SN850 dual drives. Benchmarks by Michael Larabel.

HTML result view exported from: https://openbenchmarking.org/result/2108260-PTS-STORAGET40&rdt&grs.

Btrfs RAID Benchmarks on Linux 5.14ProcessorMotherboardChipsetMemoryDiskGraphicsAudioMonitorNetworkOSKernelDesktopDisplay ServerOpenGLVulkanCompilerFile-SystemScreen ResolutionSingleRAID0RAID1AMD Ryzen 9 5900X 12-Core @ 3.70GHz (12 Cores / 24 Threads)ASUS ROG CROSSHAIR VIII HERO (3501 BIOS)AMD Starship/Matisse16GB1000GB Western Digital WD_BLACK SN850 1TB + 1000GB Western Digital WDS100T1X0E-00AFY0AMD DIMGREY_CAVEFISH 8GB (2900/1000MHz)AMD Device ab28ASUS VP28URealtek RTL8125 2.5GbE + Intel I211Ubuntu 20.045.14.0-051400rc7-generic (x86_64) 20210822GNOME Shell 3.36.9X Server 1.20.114.6 Mesa 21.3.0-devel (git-2b4b310 2021-08-21 focal-oibaf-ppa) (LLVM 12.0.1)1.2.185GCC 9.3.0btrfs3840x21602 x 1000GB Western Digital WDS100T1X0E-00AFY0 + 1000GB Western Digital WD_BLACK SN850 1TBOpenBenchmarking.orgKernel Details- Transparent Huge Pages: madviseCompiler Details- --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2 --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-9-HskZEa/gcc-9-9.3.0/debian/tmp-nvptx/usr,hsa --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Disk Details- Single: NONE / relatime,rw,space_cache,ssd,subvol=/,subvolid=5 / Block Size: 4096- RAID0: NONE / relatime,rw,space_cache,ssd,subvol=/,subvolid=5 / RAID0 Block Size: 4096- RAID1: NONE / relatime,rw,space_cache,ssd,subvol=/,subvolid=5 / RAID1 Block Size: 4096Processor Details- Scaling Governor: acpi-cpufreq ondemand (Boost: Enabled) - CPU Microcode: 0xa201009Security Details- itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl and seccomp + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Full AMD retpoline IBPB: conditional IBRS_FW STIBP: always-on RSB filling + srbds: Not affected + tsx_async_abort: Not affected

Btrfs RAID Benchmarks on Linux 5.14fio: Seq Write - IO_uring - No - Yes - 2MB - Default Test Directoryfs-mark: 5000 Files, 1MB Size, 4 Threadsior: 16MB - Default Test Directoryfs-mark: 1000 Files, 1MB Sizefs-mark: 4000 Files, 32 Sub Dirs, 1MB Sizefio: Rand Write - IO_uring - No - Yes - 4KB - Default Test Directorysqlite: 8pgbench: 1000 - 50 - Read Write - Average Latencypgbench: 1000 - 50 - Read Writepgbench: 100 - 50 - Read Write - Average Latencyfio: Rand Read - IO_uring - No - Yes - 4KB - Default Test Directorypgbench: 100 - 50 - Read Writemysqlslap: 256mysqlslap: 512dbench: 6fio: Seq Read - IO_uring - No - Yes - 2MB - Default Test Directorypgbench: 1000 - 50 - Read Onlymysqlslap: 128pgbench: 1000 - 50 - Read Only - Average LatencySingleRAID0RAID132731841.02291.05872.2859.214333324.3452.456203612.053554333243567257062005.6532533113798560.16164692617.62909.751085.91065.915866724.0692.435205392.047553000244307016872042.4133183123618600.16032381912.72454.13947.9905.213966725.7902.572194462.12535333236217106841982.6333253100298540.161OpenBenchmarking.org

Flexible IO Tester

Type: Sequential Write - Engine: IO_uring - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.25Type: Sequential Write - Engine: IO_uring - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectorySingleRAID0RAID114002800420056007000SE +/- 0.88, N = 3SE +/- 3.71, N = 3SE +/- 1.15, N = 33273646932381. (CC) gcc options: -rdynamic -ll -lnuma -lrt -lz -lpthread -lm -ldl -laio -lcurl -lssl -lcrypto -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

FS-Mark

Test: 5000 Files, 1MB Size, 4 Threads

OpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 5000 Files, 1MB Size, 4 ThreadsSingleRAID0RAID16001200180024003000SE +/- 21.56, N = 3SE +/- 5.57, N = 3SE +/- 4.42, N = 31841.02617.61912.71. (CC) gcc options: -static

IOR

Block Size: 16MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterIOR 3.3.0Block Size: 16MB - Disk Target: Default Test DirectorySingleRAID0RAID16001200180024003000SE +/- 10.10, N = 3SE +/- 9.12, N = 3SE +/- 26.32, N = 32291.052909.752454.13MIN: 1995.52 / MAX: 2735.91MIN: 2289.94 / MAX: 3517.65MIN: 2101.18 / MAX: 2865.911. (CC) gcc options: -O2 -lm -pthread -lmpi

FS-Mark

Test: 1000 Files, 1MB Size

OpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 1000 Files, 1MB SizeSingleRAID0RAID12004006008001000SE +/- 3.88, N = 3SE +/- 1.54, N = 3SE +/- 8.33, N = 3872.21085.9947.91. (CC) gcc options: -static

FS-Mark

Test: 4000 Files, 32 Sub Dirs, 1MB Size

OpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 4000 Files, 32 Sub Dirs, 1MB SizeSingleRAID0RAID12004006008001000SE +/- 2.65, N = 3SE +/- 9.58, N = 3SE +/- 7.27, N = 3859.21065.9905.21. (CC) gcc options: -static

Flexible IO Tester

Type: Random Write - Engine: IO_uring - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.25Type: Random Write - Engine: IO_uring - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectorySingleRAID0RAID130K60K90K120K150KSE +/- 1201.85, N = 3SE +/- 881.92, N = 3SE +/- 1763.83, N = 31433331586671396671. (CC) gcc options: -rdynamic -ll -lnuma -lrt -lz -lpthread -lm -ldl -laio -lcurl -lssl -lcrypto -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

SQLite

Threads / Copies: 8

OpenBenchmarking.orgSeconds, Fewer Is BetterSQLite 3.30.1Threads / Copies: 8SingleRAID0RAID1612182430SE +/- 0.31, N = 15SE +/- 0.22, N = 7SE +/- 0.15, N = 324.3524.0725.791. (CC) gcc options: -O2 -lz -lm -ldl -lpthread

PostgreSQL pgbench

Scaling Factor: 1000 - Clients: 50 - Mode: Read Write - Average Latency

OpenBenchmarking.orgms, Fewer Is BetterPostgreSQL pgbench 13.0Scaling Factor: 1000 - Clients: 50 - Mode: Read Write - Average LatencySingleRAID0RAID10.57871.15741.73612.31482.8935SE +/- 0.012, N = 3SE +/- 0.018, N = 3SE +/- 0.016, N = 32.4562.4352.5721. (CC) gcc options: -fno-strict-aliasing -fwrapv -O2 -lpgcommon -lpgport -lpq -lpthread -lrt -ldl -lm

PostgreSQL pgbench

Scaling Factor: 1000 - Clients: 50 - Mode: Read Write

OpenBenchmarking.orgTPS, More Is BetterPostgreSQL pgbench 13.0Scaling Factor: 1000 - Clients: 50 - Mode: Read WriteSingleRAID0RAID14K8K12K16K20KSE +/- 101.55, N = 3SE +/- 148.43, N = 3SE +/- 124.29, N = 32036120539194461. (CC) gcc options: -fno-strict-aliasing -fwrapv -O2 -lpgcommon -lpgport -lpq -lpthread -lrt -ldl -lm

PostgreSQL pgbench

Scaling Factor: 100 - Clients: 50 - Mode: Read Write - Average Latency

OpenBenchmarking.orgms, Fewer Is BetterPostgreSQL pgbench 13.0Scaling Factor: 100 - Clients: 50 - Mode: Read Write - Average LatencySingleRAID0RAID10.4770.9541.4311.9082.385SE +/- 0.009, N = 3SE +/- 0.013, N = 3SE +/- 0.009, N = 32.0532.0472.1201. (CC) gcc options: -fno-strict-aliasing -fwrapv -O2 -lpgcommon -lpgport -lpq -lpthread -lrt -ldl -lm

Flexible IO Tester

Type: Random Read - Engine: IO_uring - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.25Type: Random Read - Engine: IO_uring - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectorySingleRAID0RAID1120K240K360K480K600KSE +/- 2905.93, N = 3SE +/- 881.92, N = 35543335530005353331. (CC) gcc options: -rdynamic -ll -lnuma -lrt -lz -lpthread -lm -ldl -laio -lcurl -lssl -lcrypto -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

PostgreSQL pgbench

Scaling Factor: 100 - Clients: 50 - Mode: Read Write

OpenBenchmarking.orgTPS, More Is BetterPostgreSQL pgbench 13.0Scaling Factor: 100 - Clients: 50 - Mode: Read WriteSingleRAID0RAID15K10K15K20K25KSE +/- 104.55, N = 3SE +/- 155.64, N = 3SE +/- 97.06, N = 32435624430236211. (CC) gcc options: -fno-strict-aliasing -fwrapv -O2 -lpgcommon -lpgport -lpq -lpthread -lrt -ldl -lm

MariaDB

Clients: 256

OpenBenchmarking.orgQueries Per Second, More Is BetterMariaDB 10.6.4Clients: 256SingleRAID0RAID1160320480640800SE +/- 3.04, N = 3SE +/- 8.54, N = 3SE +/- 2.60, N = 37257017101. (CXX) g++ options: -pie -fPIC -fstack-protector -O3 -pthread -lbz2 -lnuma -lpcre2-8 -lcrypt -laio -lz -lm -lssl -lcrypto -lpthread -ldl

MariaDB

Clients: 512

OpenBenchmarking.orgQueries Per Second, More Is BetterMariaDB 10.6.4Clients: 512SingleRAID0RAID1150300450600750SE +/- 1.20, N = 3SE +/- 3.39, N = 3SE +/- 1.43, N = 37066876841. (CXX) g++ options: -pie -fPIC -fstack-protector -O3 -pthread -lbz2 -lnuma -lpcre2-8 -lcrypt -laio -lz -lm -lssl -lcrypto -lpthread -ldl

Dbench

Client Count: 6

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.0Client Count: 6SingleRAID0RAID1400800120016002000SE +/- 3.25, N = 3SE +/- 1.79, N = 3SE +/- 3.77, N = 32005.652042.411982.631. (CC) gcc options: -lpopt -O2

Flexible IO Tester

Type: Sequential Read - Engine: IO_uring - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.25Type: Sequential Read - Engine: IO_uring - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectorySingleRAID0RAID17001400210028003500SE +/- 39.61, N = 3SE +/- 12.44, N = 3SE +/- 21.52, N = 33253331833251. (CC) gcc options: -rdynamic -ll -lnuma -lrt -lz -lpthread -lm -ldl -laio -lcurl -lssl -lcrypto -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

PostgreSQL pgbench

Scaling Factor: 1000 - Clients: 50 - Mode: Read Only

OpenBenchmarking.orgTPS, More Is BetterPostgreSQL pgbench 13.0Scaling Factor: 1000 - Clients: 50 - Mode: Read OnlySingleRAID0RAID170K140K210K280K350KSE +/- 2881.18, N = 3SE +/- 3920.14, N = 3SE +/- 3356.29, N = 33113793123613100291. (CC) gcc options: -fno-strict-aliasing -fwrapv -O2 -lpgcommon -lpgport -lpq -lpthread -lrt -ldl -lm

MariaDB

Clients: 128

OpenBenchmarking.orgQueries Per Second, More Is BetterMariaDB 10.6.4Clients: 128SingleRAID0RAID12004006008001000SE +/- 2.40, N = 3SE +/- 11.17, N = 3SE +/- 1.90, N = 38568608541. (CXX) g++ options: -pie -fPIC -fstack-protector -O3 -pthread -lbz2 -lnuma -lpcre2-8 -lcrypt -laio -lz -lm -lssl -lcrypto -lpthread -ldl

PostgreSQL pgbench

Scaling Factor: 1000 - Clients: 50 - Mode: Read Only - Average Latency

OpenBenchmarking.orgms, Fewer Is BetterPostgreSQL pgbench 13.0Scaling Factor: 1000 - Clients: 50 - Mode: Read Only - Average LatencySingleRAID0RAID10.03620.07240.10860.14480.181SE +/- 0.002, N = 3SE +/- 0.002, N = 3SE +/- 0.002, N = 30.1610.1600.1611. (CC) gcc options: -fno-strict-aliasing -fwrapv -O2 -lpgcommon -lpgport -lpq -lpthread -lrt -ldl -lm

Phoronix Test Suite v10.8.5