FreeBSD ZFS RAID Benchmarks

AMD EPYC 7601 32-Core testing on FreeBSD via the Phoronix Test Suite.

HTML result view exported from: https://openbenchmarking.org/result/1812085-SK-1812089SK56&gru&sor.

FreeBSD ZFS RAID BenchmarksProcessorMotherboardChipsetMemoryDiskOSKernelCompilerFile-SystemScreen ResolutionRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle DiskAMD EPYC 7601 32-Core @ 2.20GHz (128 Cores)Dell PowerEdge R7425AMD [AMD] Family 17h (Models 00h-0fh) Root Complex524032MB20 x ATA Samsung SSD 860 1B6Q + DP BP14G+EXP 2.17 + SSDSCKJB120G7R N201DL43FreeBSD12.0-RC3 (x86_64)Clang 6.0.1 (SVN 335540)zfs1024x768ATA Samsung SSD 860 1B6Q + DP BP14G+EXP 2.17 + SSDSCKJB120G7R N201DL43OpenBenchmarking.orgPython Details- Python 2.7.15 + Python 3.6.6

FreeBSD ZFS RAID Benchmarksblogbench: Readblogbench: Writefio: Rand Read - POSIX AIO - No - Yes - 2MB - Default Test Directoryfio: Seq Read - POSIX AIO - No - Yes - 2MB - Default Test Directoryfio: Rand Read - POSIX AIO - No - Yes - 4KB - Default Test Directoryfio: Rand Write - POSIX AIO - No - Yes - 2MB - Default Test Directoryfio: Rand Write - POSIX AIO - No - Yes - 4KB - Default Test Directoryfio: Seq Read - POSIX AIO - No - Yes - 4KB - Default Test Directoryfio: Seq Write - POSIX AIO - No - Yes - 2MB - Default Test Directoryfio: Seq Write - POSIX AIO - No - Yes - 4KB - Default Test Directoryiozone: 1MB - 8GB - Read Performanceiozone: 4Kb - 8GB - Read Performanceiozone: 1MB - 8GB - Write Performanceiozone: 4Kb - 8GB - Write Performanceiozone: 64Kb - 8GB - Read Performanceiozone: 1MB - 512MB - Read Performanceiozone: 4Kb - 512MB - Read Performanceiozone: 64Kb - 8GB - Write Performanceiozone: 1MB - 512MB - Write Performanceiozone: 4Kb - 512MB - Write Performanceiozone: 64Kb - 512MB - Read Performanceiozone: 64Kb - 512MB - Write Performancecompilebench: Compilecompilebench: Initial Createpostmark: Disk Transaction Performancepgbench: Buffer Test - Normal Load - Read Onlypgbench: Buffer Test - Normal Load - Read Writesqlite: Timed SQLite Insertionsosbench: Create FilesRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 DisksSingle Disk16337231471920018900238812592.2523880851806639209625874456077638920851650204837159211372925136323241943191535.1476.7316181231561910019333238657690.87245699517966912107304339961076380207218031911388597213311045139331842403288026.1880.5515697030181913319133238829079.1324182071716694211125944196117644021171658167136156141342890133326041800187839.9180.071612693208184671883324019224020066462107349948860556630210922062370409569317281141140431242780214125.0277.71OpenBenchmarking.org

BlogBench

Test: Read

OpenBenchmarking.orgFinal Score, More Is BetterBlogBench 1.1Test: ReadRAIDZ1 20 DisksStripe 20 DisksSingle DiskRAIDZ3 20 Disks30K60K90K120K150KSE +/- 2001.90, N = 8SE +/- 1752.00, N = 3SE +/- 2686.95, N = 3SE +/- 1041.60, N = 31633721618121612691569701. (CC) clang options: -O2 -pthread

BlogBench

Test: Write

OpenBenchmarking.orgFinal Score, More Is BetterBlogBench 1.1Test: WriteSingle DiskStripe 20 DisksRAIDZ1 20 DisksRAIDZ3 20 Disks7001400210028003500SE +/- 52.94, N = 3SE +/- 42.15, N = 3SE +/- 90.82, N = 3SE +/- 33.71, N = 332083156314730181. (CC) clang options: -O2 -pthread

Flexible IO Tester

Type: Random Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryRAIDZ1 20 DisksRAIDZ3 20 DisksStripe 20 DisksSingle Disk4K8K12K16K20KSE +/- 120.19, N = 3SE +/- 290.59, N = 3192001913319100184671. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Random Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectorySingle DiskStripe 20 DisksRAIDZ1 20 DisksRAIDZ3 20 Disks13K26K39K52K65KSE +/- 578.31, N = 3SE +/- 916.52, N = 3SE +/- 384.42, N = 3SE +/- 472.58, N = 3612676100060933609001. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Random Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectorySingle DiskRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks11K22K33K44K55KSE +/- 33.33, N = 3SE +/- 393.70, N = 4SE +/- 120.19, N = 3SE +/- 145.30, N = 3491672360023267202671. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Sequential Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryStripe 20 DisksRAIDZ3 20 DisksRAIDZ1 20 DisksSingle Disk4K8K12K16K20KSE +/- 33.33, N = 3SE +/- 176.38, N = 3SE +/- 251.66, N = 3SE +/- 272.85, N = 3193331913318900188331. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Sequential Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryStripe 20 DisksRAIDZ3 20 DisksSingle DiskRAIDZ1 20 Disks13K26K39K52K65KSE +/- 566.67, N = 3SE +/- 145.30, N = 3SE +/- 233.33, N = 3SE +/- 470.22, N = 3627676163361433608671. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Sequential Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectorySingle DiskRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks11K22K33K44K55KSE +/- 166.67, N = 3SE +/- 145.30, N = 3SE +/- 470.22, N = 3512674613345800438331. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Random Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectorySingle DiskRAIDZ3 20 DisksStripe 20 DisksRAIDZ1 20 Disks50100150200250SE +/- 2.19, N = 3SE +/- 1.76, N = 3SE +/- 3.71, N = 3SE +/- 1.53, N = 32402382382381. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Random Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryRAIDZ3 20 DisksRAIDZ1 20 DisksStripe 20 Disks2K4K6K8K10KSE +/- 94.07, N = 12SE +/- 88.88, N = 10SE +/- 102.00, N = 128290812565761. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Random Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Random Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectorySingle DiskRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks4080120160200SE +/- 1.51, N = 4SE +/- 0.54, N = 3SE +/- 0.50, N = 3192.0092.2590.8779.131. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Sequential Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryStripe 20 DisksRAIDZ3 20 DisksSingle DiskRAIDZ1 20 Disks50100150200250SE +/- 2.00, N = 3SE +/- 0.88, N = 3SE +/- 1.67, N = 32452412402381. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Sequential Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryRAIDZ3 20 DisksRAIDZ1 20 DisksStripe 20 Disks2K4K6K8K10KSE +/- 87.61, N = 12SE +/- 72.09, N = 3SE +/- 113.76, N = 128207808569951. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

Flexible IO Tester

Type: Sequential Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.1Type: Sequential Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectorySingle DiskRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks4080120160200SE +/- 0.88, N = 3SE +/- 2.00, N = 32001801791711. (CC) clang options: -rdynamic -std=gnu99 -ffast-math -include -O3 -U_FORTIFY_SOURCE -ll -lrdmacm -libverbs -lz -lm -lpthread -lrt

IOzone

Record Size: 1MB - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 8GB - Disk Test: Read PerformanceRAIDZ3 20 DisksStripe 20 DisksSingle DiskRAIDZ1 20 Disks14002800420056007000SE +/- 26.23, N = 3SE +/- 16.35, N = 3SE +/- 24.91, N = 3SE +/- 15.45, N = 36694669166466639

IOzone

Record Size: 4Kb - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 8GB - Disk Test: Read PerformanceRAIDZ3 20 DisksSingle DiskStripe 20 DisksRAIDZ1 20 Disks5001000150020002500SE +/- 5.35, N = 3SE +/- 6.80, N = 3SE +/- 4.94, N = 3SE +/- 4.29, N = 32111210721072096

IOzone

Record Size: 1MB - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 8GB - Disk Test: Write PerformanceSingle DiskStripe 20 DisksRAIDZ3 20 DisksRAIDZ1 20 Disks8001600240032004000SE +/- 67.16, N = 3SE +/- 21.58, N = 3SE +/- 11.48, N = 3SE +/- 35.29, N = 33499304325942587

IOzone

Record Size: 4Kb - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 8GB - Disk Test: Write PerformanceSingle DiskRAIDZ1 20 DisksRAIDZ3 20 DisksStripe 20 Disks110220330440550SE +/- 7.91, N = 3SE +/- 5.97, N = 3SE +/- 5.15, N = 12SE +/- 4.31, N = 12488445419399

IOzone

Record Size: 64Kb - File Size: 8GB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 8GB - Disk Test: Read PerformanceRAIDZ3 20 DisksStripe 20 DisksRAIDZ1 20 DisksSingle Disk13002600390052006500SE +/- 20.91, N = 3SE +/- 17.33, N = 3SE +/- 19.19, N = 3SE +/- 34.98, N = 36117610760776055

IOzone

Record Size: 1MB - File Size: 512MB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 512MB - Disk Test: Read PerformanceSingle DiskRAIDZ3 20 DisksRAIDZ1 20 DisksStripe 20 Disks14002800420056007000SE +/- 26.95, N = 3SE +/- 82.98, N = 7SE +/- 93.30, N = 5SE +/- 64.15, N = 36630644063896380

IOzone

Record Size: 4Kb - File Size: 512MB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 512MB - Disk Test: Read PerformanceRAIDZ3 20 DisksSingle DiskRAIDZ1 20 DisksStripe 20 Disks5001000150020002500SE +/- 2.50, N = 3SE +/- 6.52, N = 3SE +/- 3.36, N = 3SE +/- 40.82, N = 32117210920852072

IOzone

Record Size: 64Kb - File Size: 8GB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 8GB - Disk Test: Write PerformanceSingle DiskStripe 20 DisksRAIDZ3 20 DisksRAIDZ1 20 Disks5001000150020002500SE +/- 29.66, N = 3SE +/- 17.15, N = 3SE +/- 5.37, N = 3SE +/- 27.14, N = 32206180316581650

IOzone

Record Size: 1MB - File Size: 512MB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 1MB - File Size: 512MB - Disk Test: Write PerformanceSingle DiskRAIDZ1 20 DisksStripe 20 DisksRAIDZ3 20 Disks5001000150020002500SE +/- 32.40, N = 12SE +/- 22.20, N = 3SE +/- 139.65, N = 12SE +/- 31.70, N = 32370204819111671

IOzone

Record Size: 4Kb - File Size: 512MB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 4Kb - File Size: 512MB - Disk Test: Write PerformanceSingle DiskStripe 20 DisksRAIDZ1 20 DisksRAIDZ3 20 Disks90180270360450SE +/- 5.91, N = 12SE +/- 4.40, N = 3SE +/- 5.25, N = 12SE +/- 4.33, N = 8409388371361

IOzone

Record Size: 64Kb - File Size: 512MB - Disk Test: Read Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 512MB - Disk Test: Read PerformanceStripe 20 DisksRAIDZ1 20 DisksSingle DiskRAIDZ3 20 Disks13002600390052006500SE +/- 37.94, N = 3SE +/- 55.94, N = 3SE +/- 128.37, N = 12SE +/- 197.26, N = 125972592156935614

IOzone

Record Size: 64Kb - File Size: 512MB - Disk Test: Write Performance

OpenBenchmarking.orgMB/s, More Is BetterIOzone 3.465Record Size: 64Kb - File Size: 512MB - Disk Test: Write PerformanceSingle DiskRAIDZ1 20 DisksRAIDZ3 20 DisksStripe 20 Disks400800120016002000SE +/- 19.26, N = 12SE +/- 22.97, N = 12SE +/- 17.84, N = 12SE +/- 17.14, N = 121728137213421331

Compile Bench

Test: Compile

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: CompileSingle DiskStripe 20 DisksRAIDZ1 20 DisksRAIDZ3 20 Disks2004006008001000SE +/- 6.65, N = 3SE +/- 13.52, N = 3SE +/- 15.82, N = 4SE +/- 5.79, N = 311411045925890

Compile Bench

Test: Initial Create

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Initial CreateSingle DiskStripe 20 DisksRAIDZ1 20 DisksRAIDZ3 20 Disks306090120150SE +/- 0.48, N = 3SE +/- 0.07, N = 3SE +/- 0.55, N = 3SE +/- 0.56, N = 3140139136133

PostMark

Disk Transaction Performance

OpenBenchmarking.orgTPS, More Is BetterPostMark 1.51Disk Transaction PerformanceSingle DiskStripe 20 DisksRAIDZ3 20 DisksRAIDZ1 20 Disks9001800270036004500SE +/- 99.00, N = 9SE +/- 14.67, N = 3SE +/- 14.33, N = 3SE +/- 13.67, N = 343123318326032321. (CC) clang options: -O3

PostgreSQL pgbench

Scaling: Buffer Test - Test: Normal Load - Mode: Read Only

OpenBenchmarking.orgTPS, More Is BetterPostgreSQL pgbench 10.3Scaling: Buffer Test - Test: Normal Load - Mode: Read OnlySingle DiskStripe 20 DisksRAIDZ1 20 DisksRAIDZ3 20 Disks9K18K27K36K45KSE +/- 280.54, N = 3SE +/- 259.57, N = 3SE +/- 113.65, N = 3SE +/- 64.59, N = 3427804240341943418001. (CC) clang options: -fno-strict-aliasing -fwrapv -O2 -fPIC -lpgport -shared

PostgreSQL pgbench

Scaling: Buffer Test - Test: Normal Load - Mode: Read Write

OpenBenchmarking.orgTPS, More Is BetterPostgreSQL pgbench 10.3Scaling: Buffer Test - Test: Normal Load - Mode: Read WriteStripe 20 DisksSingle DiskRAIDZ1 20 DisksRAIDZ3 20 Disks6001200180024003000SE +/- 68.03, N = 8SE +/- 32.07, N = 9SE +/- 24.99, N = 7SE +/- 32.38, N = 328802141191518781. (CC) clang options: -fno-strict-aliasing -fwrapv -O2 -fPIC -lpgport -shared

SQLite

Timed SQLite Insertions

OpenBenchmarking.orgSeconds, Fewer Is BetterSQLite 3.22Timed SQLite InsertionsSingle DiskStripe 20 DisksRAIDZ1 20 DisksRAIDZ3 20 Disks918273645SE +/- 0.03, N = 3SE +/- 0.17, N = 3SE +/- 0.55, N = 3SE +/- 0.27, N = 325.0226.1835.1439.911. (CC) clang options: -O2 -lz -lpthread

OSBench

Test: Create Files

OpenBenchmarking.orgus Per Event, Fewer Is BetterOSBenchTest: Create FilesRAIDZ1 20 DisksSingle DiskRAIDZ3 20 DisksStripe 20 Disks20406080100SE +/- 0.45, N = 3SE +/- 0.78, N = 3SE +/- 1.37, N = 3SE +/- 0.91, N = 376.7377.7180.0780.551. (CC) clang options: -lm

Phoronix Test Suite v10.8.4