FreeBSD ZFS RAID Benchmarks

AMD EPYC 7601 32-Core testing on FreeBSD via the Phoronix Test Suite.

HTML result view exported from: https://openbenchmarking.org/result/1812085-SK-1812089SK56&grt.

FreeBSD ZFS RAID BenchmarksProcessorMotherboardChipsetMemoryDiskOSKernelCompilerFile-SystemScreen ResolutionRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle DiskAMD EPYC 7601 32-Core @ 2.20GHz (128 Cores)Dell PowerEdge R7425AMD [AMD] Family 17h (Models 00h-0fh) Root Complex524032MB20 x ATA Samsung SSD 860 1B6Q + DP BP14G+EXP 2.17 + SSDSCKJB120G7R N201DL43FreeBSD12.0-RC3 (x86_64)Clang 6.0.1 (SVN 335540)zfs1024x768ATA Samsung SSD 860 1B6Q + DP BP14G+EXP 2.17 + SSDSCKJB120G7R N201DL43OpenBenchmarking.orgPython Details- Python 2.7.15 + Python 3.6.6

FreeBSD ZFS RAID Benchmarksblogbench: Readblogbench: Writecompilebench: Compilecompilebench: Initial Createfio: Rand Read - POSIX AIO - No - Yes - 2MB - Default Test Directoryfio: Rand Read - POSIX AIO - No - Yes - 4KB - Default Test Directoryfio: Rand Write - POSIX AIO - No - Yes - 2MB - Default Test Directoryfio: Rand Write - POSIX AIO - No - Yes - 4KB - Default Test Directoryfio: Seq Read - POSIX AIO - No - Yes - 2MB - Default Test Directoryfio: Seq Read - POSIX AIO - No - Yes - 4KB - Default Test Directoryfio: Seq Write - POSIX AIO - No - Yes - 2MB - Default Test Directoryfio: Seq Write - POSIX AIO - No - Yes - 4KB - Default Test Directoryiozone: 1MB - 8GB - Read Performanceiozone: 4Kb - 8GB - Read Performanceiozone: 1MB - 8GB - Write Performanceiozone: 4Kb - 8GB - Write Performanceiozone: 64Kb - 8GB - Read Performanceiozone: 1MB - 512MB - Read Performanceiozone: 4Kb - 512MB - Read Performanceiozone: 64Kb - 8GB - Write Performanceiozone: 1MB - 512MB - Write Performanceiozone: 4Kb - 512MB - Write Performanceiozone: 64Kb - 512MB - Read Performanceiozone: 64Kb - 512MB - Write Performanceosbench: Create Filespgbench: Buffer Test - Normal Load - Read Onlypgbench: Buffer Test - Normal Load - Read Writepostmark: Disk Transaction Performancesqlite: Timed SQLite InsertionsRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk163372314792513619200238812592.25189002388085180663920962587445607763892085165020483715921137276.73419431915323235.141618123156104513919100238657690.87193332456995179669121073043399610763802072180319113885972133180.55424032880331826.18156970301889013319133238829079.13191332418207171669421112594419611764402117165816713615614134280.07418001878326039.91161269320811411401846724019218833240200664621073499488605566302109220623704095693172877.71427802141431225.02OpenBenchmarking.org

BlogBench

Test: Read

OpenBenchmarking.orgFinal Score, More Is BetterBlogBench 1.1Test: ReadRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk30K60K90K120K150KSE +/- 2001.90, N = 8SE +/- 1752.00, N = 3SE +/- 1041.60, N = 3SE +/- 2686.95, N = 31633721618121569701612691. (CC) clang options: -O2 -pthread

BlogBench

Test: Write

OpenBenchmarking.orgFinal Score, More Is BetterBlogBench 1.1Test: WriteRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk7001400210028003500SE +/- 90.82, N = 3SE +/- 42.15, N = 3SE +/- 33.71, N = 3SE +/- 52.94, N = 331473156301832081. (CC) clang options: -O2 -pthread

Compile Bench

Test: Compile

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: CompileRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk2004006008001000SE +/- 15.82, N = 4SE +/- 13.52, N = 3SE +/- 5.79, N = 3SE +/- 6.65, N = 392510458901141

Compile Bench

Test: Initial Create

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Initial CreateRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk306090120150SE +/- 0.55, N = 3SE +/- 0.07, N = 3SE +/- 0.56, N = 3SE +/- 0.48, N = 3136139133140

Flexible IO Tester

Type: Random Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk4K8K12K16K20KSE +/- 120.19, N = 3SE +/- 290.59, N = 3192001910019133184671. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Random Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk50100150200250SE +/- 1.53, N = 3SE +/- 3.71, N = 3SE +/- 1.76, N = 3SE +/- 2.19, N = 32382382382401. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Random Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk13K26K39K52K65KSE +/- 384.42, N = 3SE +/- 916.52, N = 3SE +/- 472.58, N = 3SE +/- 578.31, N = 3609336100060900612671. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Random Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks2K4K6K8K10KSE +/- 88.88, N = 10SE +/- 102.00, N = 12SE +/- 94.07, N = 128125657682901. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Random Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk4080120160200SE +/- 1.51, N = 4SE +/- 0.54, N = 3SE +/- 0.50, N = 392.2590.8779.13192.001. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Random Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk11K22K33K44K55KSE +/- 393.70, N = 4SE +/- 120.19, N = 3SE +/- 145.30, N = 3SE +/- 33.33, N = 3236002326720267491671. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Sequential Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk4K8K12K16K20KSE +/- 251.66, N = 3SE +/- 33.33, N = 3SE +/- 176.38, N = 3SE +/- 272.85, N = 3189001933319133188331. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Sequential Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk50100150200250SE +/- 1.67, N = 3SE +/- 2.00, N = 3SE +/- 0.88, N = 32382452412401. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Sequential Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk13K26K39K52K65KSE +/- 470.22, N = 3SE +/- 566.67, N = 3SE +/- 145.30, N = 3SE +/- 233.33, N = 3608676276761633614331. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Sequential Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks2K4K6K8K10KSE +/- 72.09, N = 3SE +/- 113.76, N = 12SE +/- 87.61, N = 128085699582071. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Sequential Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk4080120160200SE +/- 2.00, N = 3SE +/- 0.88, N = 31801791712001. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Sequential Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk11K22K33K44K55KSE +/- 145.30, N = 3SE +/- 470.22, N = 3SE +/- 166.67, N = 3461334580043833512671. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

IOzone

Record Size: 1MB - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 8GB - Disk Test: Read PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk14002800420056007000SE +/- 15.45, N = 3SE +/- 16.35, N = 3SE +/- 26.23, N = 3SE +/- 24.91, N = 36639669166946646

IOzone

Record Size: 4Kb - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 8GB - Disk Test: Read PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk5001000150020002500SE +/- 4.29, N = 3SE +/- 4.94, N = 3SE +/- 5.35, N = 3SE +/- 6.80, N = 32096210721112107

IOzone

Record Size: 1MB - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 8GB - Disk Test: Write PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk8001600240032004000SE +/- 35.29, N = 3SE +/- 21.58, N = 3SE +/- 11.48, N = 3SE +/- 67.16, N = 32587304325943499

IOzone

Record Size: 4Kb - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 8GB - Disk Test: Write PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk110220330440550SE +/- 5.97, N = 3SE +/- 4.31, N = 12SE +/- 5.15, N = 12SE +/- 7.91, N = 3445399419488

IOzone

Record Size: 64Kb - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 8GB - Disk Test: Read PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk13002600390052006500SE +/- 19.19, N = 3SE +/- 17.33, N = 3SE +/- 20.91, N = 3SE +/- 34.98, N = 36077610761176055

IOzone

Record Size: 1MB - File Size: 512MB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 512MB - Disk Test: Read PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk14002800420056007000SE +/- 93.30, N = 5SE +/- 64.15, N = 3SE +/- 82.98, N = 7SE +/- 26.95, N = 36389638064406630

IOzone

Record Size: 4Kb - File Size: 512MB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 512MB - Disk Test: Read PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk5001000150020002500SE +/- 3.36, N = 3SE +/- 40.82, N = 3SE +/- 2.50, N = 3SE +/- 6.52, N = 32085207221172109

IOzone

Record Size: 64Kb - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 8GB - Disk Test: Write PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk5001000150020002500SE +/- 27.14, N = 3SE +/- 17.15, N = 3SE +/- 5.37, N = 3SE +/- 29.66, N = 31650180316582206

IOzone

Record Size: 1MB - File Size: 512MB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 512MB - Disk Test: Write PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk5001000150020002500SE +/- 22.20, N = 3SE +/- 139.65, N = 12SE +/- 31.70, N = 3SE +/- 32.40, N = 122048191116712370

IOzone

Record Size: 4Kb - File Size: 512MB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 512MB - Disk Test: Write PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk90180270360450SE +/- 5.25, N = 12SE +/- 4.40, N = 3SE +/- 4.33, N = 8SE +/- 5.91, N = 12371388361409

IOzone

Record Size: 64Kb - File Size: 512MB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 512MB - Disk Test: Read PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk13002600390052006500SE +/- 55.94, N = 3SE +/- 37.94, N = 3SE +/- 197.26, N = 12SE +/- 128.37, N = 125921597256145693

IOzone

Record Size: 64Kb - File Size: 512MB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 512MB - Disk Test: Write PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk400800120016002000SE +/- 22.97, N = 12SE +/- 17.14, N = 12SE +/- 17.84, N = 12SE +/- 19.26, N = 121372133113421728

OSBench

Test: Create Files

OpenBenchmarking.orgus Per Event, Fewer Is BetterOSBenchTest: Create FilesRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk20406080100SE +/- 0.45, N = 3SE +/- 0.91, N = 3SE +/- 1.37, N = 3SE +/- 0.78, N = 376.7380.5580.0777.711. (CC) clang options: -lm

PostgreSQL pgbench

Scaling: Buffer Test - Test: Normal Load - Mode: Read Only

OpenBenchmarking.orgTPS, More Is BetterPostgreSQL pgbench 10.3Scaling: Buffer Test - Test: Normal Load - Mode: Read OnlyRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk9K18K27K36K45KSE +/- 113.65, N = 3SE +/- 259.57, N = 3SE +/- 64.59, N = 3SE +/- 280.54, N = 3419434240341800427801. (CC) clang options: -fno-strict-aliasing -fwrapv -O2 -fPIC -lpgport -shared

PostgreSQL pgbench

Scaling: Buffer Test - Test: Normal Load - Mode: Read Write

OpenBenchmarking.orgTPS, More Is BetterPostgreSQL pgbench 10.3Scaling: Buffer Test - Test: Normal Load - Mode: Read WriteRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk6001200180024003000SE +/- 24.99, N = 7SE +/- 68.03, N = 8SE +/- 32.38, N = 3SE +/- 32.07, N = 919152880187821411. (CC) clang options: -fno-strict-aliasing -fwrapv -O2 -fPIC -lpgport -shared

PostMark

Disk Transaction Performance

OpenBenchmarking.orgTPS, More Is BetterPostMark 1.51Disk Transaction PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk9001800270036004500SE +/- 13.67, N = 3SE +/- 14.67, N = 3SE +/- 14.33, N = 3SE +/- 99.00, N = 932323318326043121. (CC) clang options: -O3

SQLite

Timed SQLite Insertions

OpenBenchmarking.orgSeconds, Fewer Is BetterSQLite 3.22Timed SQLite InsertionsRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk918273645SE +/- 0.55, N = 3SE +/- 0.17, N = 3SE +/- 0.27, N = 3SE +/- 0.03, N = 335.1426.1839.9125.021. (CC) clang options: -O2 -lz -lpthread

Phoronix Test Suite v10.8.4