FreeBSD ZFS RAID Benchmarks

AMD EPYC 7601 32-Core testing on FreeBSD via the Phoronix Test Suite.

HTML result view exported from: https://openbenchmarking.org/result/1812089-SK-1812073SK81.

FreeBSD ZFS RAID BenchmarksProcessorMotherboardChipsetMemoryDiskOSKernelCompilerFile-SystemScreen ResolutionRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksAMD EPYC 7601 32-Core @ 2.20GHz (128 Cores)Dell PowerEdge R7425AMD [AMD] Family 17h (Models 00h-0fh) Root Complex524032MB20 x ATA Samsung SSD 860 1B6Q + DP BP14G+EXP 2.17 + SSDSCKJB120G7R N201DL43FreeBSD12.0-RC3 (x86_64)Clang 6.0.1 (SVN 335540)zfs1024x768OpenBenchmarking.orgPython Details- Python 2.7.15 + Python 3.6.6

FreeBSD ZFS RAID Benchmarkssqlite: Timed SQLite Insertionsfio: Rand Read - POSIX AIO - No - Yes - 2MB - Default Test Directoryfio: Rand Read - POSIX AIO - No - Yes - 4KB - Default Test Directoryfio: Rand Write - POSIX AIO - No - Yes - 2MB - Default Test Directoryfio: Rand Write - POSIX AIO - No - Yes - 4KB - Default Test Directoryfio: Seq Read - POSIX AIO - No - Yes - 2MB - Default Test Directoryfio: Seq Read - POSIX AIO - No - Yes - 4KB - Default Test Directoryfio: Seq Write - POSIX AIO - No - Yes - 2MB - Default Test Directoryfio: Seq Write - POSIX AIO - No - Yes - 4KB - Default Test Directoryblogbench: Readblogbench: Writeiozone: 1MB - 8GB - Read Performanceiozone: 4Kb - 8GB - Read Performanceiozone: 1MB - 8GB - Write Performanceiozone: 4Kb - 8GB - Write Performanceiozone: 64Kb - 8GB - Read Performanceiozone: 1MB - 512MB - Read Performanceiozone: 4Kb - 512MB - Read Performanceiozone: 64Kb - 8GB - Write Performanceiozone: 1MB - 512MB - Write Performanceiozone: 4Kb - 512MB - Write Performanceiozone: 64Kb - 512MB - Read Performanceiozone: 64Kb - 512MB - Write Performancecompilebench: Compilecompilebench: Initial Createpostmark: Disk Transaction Performancepgbench: Buffer Test - Normal Load - Read Onlypgbench: Buffer Test - Normal Load - Read Writeosbench: Create FilesRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks35.1419200238812592.2518900238808518016337231476639209625874456077638920851650204837159211372925136323241943191576.7326.1819100238657690.87193332456995179161812315666912107304339961076380207218031911388597213311045139331842403288080.5539.9119133238829079.1319133241820717115697030186694211125944196117644021171658167136156141342890133326041800187880.07OpenBenchmarking.org

SQLite

Timed SQLite Insertions

OpenBenchmarking.orgSeconds, Fewer Is BetterSQLite 3.22Timed SQLite InsertionsRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks918273645SE +/- 0.55, N = 3SE +/- 0.17, N = 3SE +/- 0.27, N = 335.1426.1839.911. (CC) clang options: -O2 -lz -lpthread

Flexible IO Tester

Type: Random Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks4K8K12K16K20KSE +/- 120.19, N = 31920019100191331. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Random Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks50100150200250SE +/- 1.53, N = 3SE +/- 3.71, N = 3SE +/- 1.76, N = 32382382381. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Random Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks13K26K39K52K65KSE +/- 384.42, N = 3SE +/- 916.52, N = 3SE +/- 472.58, N = 36093361000609001. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Random Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks2K4K6K8K10KSE +/- 88.88, N = 10SE +/- 102.00, N = 12SE +/- 94.07, N = 128125657682901. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Random Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks20406080100SE +/- 1.51, N = 4SE +/- 0.54, N = 3SE +/- 0.50, N = 392.2590.8779.131. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Random Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks5K10K15K20K25KSE +/- 393.70, N = 4SE +/- 120.19, N = 3SE +/- 145.30, N = 32360023267202671. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Sequential Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks4K8K12K16K20KSE +/- 251.66, N = 3SE +/- 33.33, N = 3SE +/- 176.38, N = 31890019333191331. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Sequential Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks50100150200250SE +/- 1.67, N = 3SE +/- 2.00, N = 32382452411. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Sequential Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks13K26K39K52K65KSE +/- 470.22, N = 3SE +/- 566.67, N = 3SE +/- 145.30, N = 36086762767616331. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Sequential Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks2K4K6K8K10KSE +/- 72.09, N = 3SE +/- 113.76, N = 12SE +/- 87.61, N = 128085699582071. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Sequential Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks4080120160200SE +/- 2.00, N = 31801791711. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Sequential Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks10K20K30K40K50KSE +/- 145.30, N = 3SE +/- 470.22, N = 34613345800438331. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

BlogBench

Test: Read

OpenBenchmarking.orgFinal Score, More Is BetterBlogBench 1.1Test: ReadRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks30K60K90K120K150KSE +/- 2001.90, N = 8SE +/- 1752.00, N = 3SE +/- 1041.60, N = 31633721618121569701. (CC) clang options: -O2 -pthread

BlogBench

Test: Write

OpenBenchmarking.orgFinal Score, More Is BetterBlogBench 1.1Test: WriteRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks7001400210028003500SE +/- 90.82, N = 3SE +/- 42.15, N = 3SE +/- 33.71, N = 33147315630181. (CC) clang options: -O2 -pthread

IOzone

Record Size: 1MB - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 8GB - Disk Test: Read PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks14002800420056007000SE +/- 15.45, N = 3SE +/- 16.35, N = 3SE +/- 26.23, N = 3663966916694

IOzone

Record Size: 4Kb - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 8GB - Disk Test: Read PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks5001000150020002500SE +/- 4.29, N = 3SE +/- 4.94, N = 3SE +/- 5.35, N = 3209621072111

IOzone

Record Size: 1MB - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 8GB - Disk Test: Write PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks7001400210028003500SE +/- 35.29, N = 3SE +/- 21.58, N = 3SE +/- 11.48, N = 3258730432594

IOzone

Record Size: 4Kb - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 8GB - Disk Test: Write PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks100200300400500SE +/- 5.97, N = 3SE +/- 4.31, N = 12SE +/- 5.15, N = 12445399419

IOzone

Record Size: 64Kb - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 8GB - Disk Test: Read PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks13002600390052006500SE +/- 19.19, N = 3SE +/- 17.33, N = 3SE +/- 20.91, N = 3607761076117

IOzone

Record Size: 1MB - File Size: 512MB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 512MB - Disk Test: Read PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks14002800420056007000SE +/- 93.30, N = 5SE +/- 64.15, N = 3SE +/- 82.98, N = 7638963806440

IOzone

Record Size: 4Kb - File Size: 512MB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 512MB - Disk Test: Read PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks5001000150020002500SE +/- 3.36, N = 3SE +/- 40.82, N = 3SE +/- 2.50, N = 3208520722117

IOzone

Record Size: 64Kb - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 8GB - Disk Test: Write PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks400800120016002000SE +/- 27.14, N = 3SE +/- 17.15, N = 3SE +/- 5.37, N = 3165018031658

IOzone

Record Size: 1MB - File Size: 512MB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 512MB - Disk Test: Write PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks400800120016002000SE +/- 22.20, N = 3SE +/- 139.65, N = 12SE +/- 31.70, N = 3204819111671

IOzone

Record Size: 4Kb - File Size: 512MB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 512MB - Disk Test: Write PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks80160240320400SE +/- 5.25, N = 12SE +/- 4.40, N = 3SE +/- 4.33, N = 8371388361

IOzone

Record Size: 64Kb - File Size: 512MB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 512MB - Disk Test: Read PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks13002600390052006500SE +/- 55.94, N = 3SE +/- 37.94, N = 3SE +/- 197.26, N = 12592159725614

IOzone

Record Size: 64Kb - File Size: 512MB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 512MB - Disk Test: Write PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks30060090012001500SE +/- 22.97, N = 12SE +/- 17.14, N = 12SE +/- 17.84, N = 12137213311342

Compile Bench

Test: Compile

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: CompileRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks2004006008001000SE +/- 15.82, N = 4SE +/- 13.52, N = 3SE +/- 5.79, N = 39251045890

Compile Bench

Test: Initial Create

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Initial CreateRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks306090120150SE +/- 0.55, N = 3SE +/- 0.07, N = 3SE +/- 0.56, N = 3136139133

PostMark

Disk Transaction Performance

OpenBenchmarking.orgTPS, More Is BetterPostMark 1.51Disk Transaction PerformanceRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks7001400210028003500SE +/- 13.67, N = 3SE +/- 14.67, N = 3SE +/- 14.33, N = 33232331832601. (CC) clang options: -O3

PostgreSQL pgbench

Scaling: Buffer Test - Test: Normal Load - Mode: Read Only

OpenBenchmarking.orgTPS, More Is BetterPostgreSQL pgbench 10.3Scaling: Buffer Test - Test: Normal Load - Mode: Read OnlyRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks9K18K27K36K45KSE +/- 113.65, N = 3SE +/- 259.57, N = 3SE +/- 64.59, N = 34194342403418001. (CC) clang options: -fno-strict-aliasing -fwrapv -O2 -fPIC -lpgport -shared

PostgreSQL pgbench

Scaling: Buffer Test - Test: Normal Load - Mode: Read Write

OpenBenchmarking.orgTPS, More Is BetterPostgreSQL pgbench 10.3Scaling: Buffer Test - Test: Normal Load - Mode: Read WriteRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks6001200180024003000SE +/- 24.99, N = 7SE +/- 68.03, N = 8SE +/- 32.38, N = 31915288018781. (CC) clang options: -fno-strict-aliasing -fwrapv -O2 -fPIC -lpgport -shared

OSBench

Test: Create Files

OpenBenchmarking.orgus Per Event, Fewer Is BetterOSBenchTest: Create FilesRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks20406080100SE +/- 0.45, N = 3SE +/- 0.91, N = 3SE +/- 1.37, N = 376.7380.5580.071. (CC) clang options: -lm

Phoronix Test Suite v10.8.4