pts-disk-different-nvmes AMD Ryzen Threadripper 1900X 8-Core testing with a Gigabyte X399 DESIGNARE EX-CF (F13a BIOS) and NVIDIA Quadro P400 on Debian 11 via the Phoronix Test Suite.
HTML result view exported from: https://openbenchmarking.org/result/2211133-NE-PTSDISKDI69&gru&sor .
pts-disk-different-nvmes Processor Motherboard Chipset Memory Disk Graphics Audio Monitor Network OS Kernel Compiler File-System Screen Resolution ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME ext4 Crucial P5 Plus 1TB NVME ZFS zraid1 8xNVME Pool ZFS zraid1 8xNVME Pool no Compression ZFS mirror 8xNVME Pool AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads) Gigabyte X399 DESIGNARE EX-CF (F13a BIOS) AMD 17h 64GB Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8 NVIDIA Quadro P400 NVIDIA GP107GL HD Audio DELL S2340T 4 x Intel I350 + Intel 8265 / 8275 Debian 11 5.10.0-19-amd64 (x86_64) GCC 10.2.1 20210110 zfs 1920x1080 ext4 zfs OpenBenchmarking.org Kernel Details - Transparent Huge Pages: always Compiler Details - --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-mutex --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-10-Km9U7s/gcc-10-10.2.1/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-10-Km9U7s/gcc-10-10.2.1/debian/tmp-gcn/usr,hsa --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details - Scaling Governor: acpi-cpufreq schedutil (Boost: Enabled) - CPU Microcode: 0x8001137 Disk Scheduler Details - ZFS zraid1 4xNVME Pool, ZFS zraid1 8xNVME Pool, ZFS zraid1 8xNVME Pool no Compression, ZFS mirror 8xNVME Pool: NONE Python Details - Python 3.9.2 Security Details - itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Mitigation of untrained return thunk; SMT vulnerable + spec_store_bypass: Mitigation of SSB disabled via prctl and seccomp + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional STIBP: disabled RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected Disk Details - ext4 mdadm raid5 4xNVME: NONE / relatime,rw,stripe=384 / raid5 nvme4n1p1[4] nvme3n1p1[2] nvme2n1p1[1] nvme1n1p1[0] Block Size: 4096 - ext4 Crucial P5 Plus 1TB NVME: NONE / relatime,rw / Block Size: 4096
pts-disk-different-nvmes fs-mark: 1000 Files, 1MB Size fs-mark: 5000 Files, 1MB Size, 4 Threads fs-mark: 4000 Files, 32 Sub Dirs, 1MB Size fs-mark: 1000 Files, 1MB Size, No Sync/FSync fio: Rand Read - Linux AIO - No - Yes - 2MB - Default Test Directory fio: Rand Read - Linux AIO - No - Yes - 4KB - Default Test Directory fio: Rand Write - Linux AIO - No - Yes - 2MB - Default Test Directory fio: Rand Write - Linux AIO - No - Yes - 4KB - Default Test Directory fio: Seq Read - Linux AIO - No - Yes - 2MB - Default Test Directory fio: Seq Read - Linux AIO - No - Yes - 4KB - Default Test Directory fio: Seq Write - Linux AIO - No - Yes - 2MB - Default Test Directory fio: Seq Write - Linux AIO - No - Yes - 4KB - Default Test Directory fio: Rand Read - Linux AIO - No - Yes - 2MB - Default Test Directory fio: Rand Read - Linux AIO - No - Yes - 4KB - Default Test Directory fio: Rand Write - Linux AIO - No - Yes - 2MB - Default Test Directory fio: Rand Write - Linux AIO - No - Yes - 4KB - Default Test Directory fio: Seq Read - Linux AIO - No - Yes - 2MB - Default Test Directory fio: Seq Read - Linux AIO - No - Yes - 4KB - Default Test Directory fio: Seq Write - Linux AIO - No - Yes - 2MB - Default Test Directory fio: Seq Write - Linux AIO - No - Yes - 4KB - Default Test Directory dbench: 12 Clients dbench: 1 Clients compilebench: Compile compilebench: Initial Create compilebench: Read Compiled Tree postmark: Disk Transaction Performance sqlite: 1 sqlite: 8 sqlite: 32 sqlite: 64 sqlite: 128 ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME ext4 Crucial P5 Plus 1TB NVME ZFS zraid1 8xNVME Pool ZFS zraid1 8xNVME Pool no Compression ZFS mirror 8xNVME Pool 646.5 1632.7 684.2 1359.3 2119 49567 1225 41500 2034 329333 1029 171333 4245 194 2458 162 4075 1287 2065 670 2626.03 410.171 1398.57 222.04 1212.73 3289 7.852 11.267 23.191 37.448 76.088 254.0 537.9 284.7 1733.3 6666 306667 476 82967 6706 221333 650 115000 1199 958 324 865 1308 449 2484.85 453.170 1478.00 411.07 2642.03 5068 8.810 20.005 31.793 45.720 80.145 584.1 522.1 435.0 1791.6 1765 329667 1623 224667 1765 335667 1614 228000 3538 1286 3252 878 3538 1311 3236 891 658.794 102.985 1483.07 423.67 2747.01 5137 8.367 15.767 27.578 40.979 59.890 646.7 1655.6 644.7 1340.7 2015 57433 1327 45367 1999 326333 1186 170667 4037 224 2661 177 4005 1275 2381 665 2472.42 380.354 1390.71 221.83 1217.80 3275 8.380 12.857 26.999 40.301 78.153 589.7 1240.0 592.1 1160.9 2081 56167 1337 48200 1995 331000 1220 174667 4170 219 2681 188 3998 1294 2446 682 2783.64 458.913 1247.36 206.77 1694.75 3275 8.379 12.827 26.892 40.003 77.047 581.1 1711.4 658.2 1338.4 3305 224333 1783 68333 2073 332000 1774 197333 6617 877 3573 267 4154 1296 3555 771 2730.75 435.704 1408.42 216.21 1454.31 3318 7.252 9.043 17.744 36.598 77.880 OpenBenchmarking.org
FS-Mark Test: 1000 Files, 1MB Size OpenBenchmarking.org Files/s, More Is Better FS-Mark 3.3 Test: 1000 Files, 1MB Size ZFS zraid1 8xNVME Pool ZFS zraid1 4xNVME Pool ZFS zraid1 8xNVME Pool no Compression ext4 Crucial P5 Plus 1TB NVME ZFS mirror 8xNVME Pool ext4 mdadm raid5 4xNVME 140 280 420 560 700 SE +/- 1.87, N = 3 SE +/- 5.49, N = 15 SE +/- 6.13, N = 3 SE +/- 13.64, N = 12 SE +/- 4.66, N = 3 SE +/- 9.50, N = 15 646.7 646.5 589.7 584.1 581.1 254.0
FS-Mark Test: 5000 Files, 1MB Size, 4 Threads OpenBenchmarking.org Files/s, More Is Better FS-Mark 3.3 Test: 5000 Files, 1MB Size, 4 Threads ZFS mirror 8xNVME Pool ZFS zraid1 8xNVME Pool ZFS zraid1 4xNVME Pool ZFS zraid1 8xNVME Pool no Compression ext4 mdadm raid5 4xNVME ext4 Crucial P5 Plus 1TB NVME 400 800 1200 1600 2000 SE +/- 2.79, N = 3 SE +/- 6.18, N = 3 SE +/- 4.95, N = 3 SE +/- 2.97, N = 3 SE +/- 4.55, N = 12 SE +/- 125.73, N = 9 1711.4 1655.6 1632.7 1240.0 537.9 522.1
FS-Mark Test: 4000 Files, 32 Sub Dirs, 1MB Size OpenBenchmarking.org Files/s, More Is Better FS-Mark 3.3 Test: 4000 Files, 32 Sub Dirs, 1MB Size ZFS zraid1 4xNVME Pool ZFS mirror 8xNVME Pool ZFS zraid1 8xNVME Pool ZFS zraid1 8xNVME Pool no Compression ext4 Crucial P5 Plus 1TB NVME ext4 mdadm raid5 4xNVME 150 300 450 600 750 SE +/- 7.90, N = 3 SE +/- 5.50, N = 15 SE +/- 3.56, N = 3 SE +/- 3.57, N = 3 SE +/- 16.63, N = 12 SE +/- 2.79, N = 3 684.2 658.2 644.7 592.1 435.0 284.7
FS-Mark Test: 1000 Files, 1MB Size, No Sync/FSync OpenBenchmarking.org Files/s, More Is Better FS-Mark 3.3 Test: 1000 Files, 1MB Size, No Sync/FSync ext4 Crucial P5 Plus 1TB NVME ext4 mdadm raid5 4xNVME ZFS zraid1 4xNVME Pool ZFS zraid1 8xNVME Pool ZFS mirror 8xNVME Pool ZFS zraid1 8xNVME Pool no Compression 400 800 1200 1600 2000 SE +/- 16.52, N = 3 SE +/- 19.84, N = 4 SE +/- 12.01, N = 15 SE +/- 10.88, N = 15 SE +/- 7.95, N = 3 SE +/- 13.85, N = 4 1791.6 1733.3 1359.3 1340.7 1338.4 1160.9
Flexible IO Tester Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory ext4 mdadm raid5 4xNVME ZFS mirror 8xNVME Pool ZFS zraid1 4xNVME Pool ZFS zraid1 8xNVME Pool no Compression ZFS zraid1 8xNVME Pool ext4 Crucial P5 Plus 1TB NVME 1400 2800 4200 5600 7000 SE +/- 14.99, N = 3 SE +/- 12.06, N = 3 SE +/- 29.49, N = 3 SE +/- 20.66, N = 3 SE +/- 6.57, N = 3 6666 3305 2119 2081 2015 1765 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory ext4 Crucial P5 Plus 1TB NVME ext4 mdadm raid5 4xNVME ZFS mirror 8xNVME Pool ZFS zraid1 8xNVME Pool ZFS zraid1 8xNVME Pool no Compression ZFS zraid1 4xNVME Pool 70K 140K 210K 280K 350K SE +/- 881.92, N = 3 SE +/- 1333.33, N = 3 SE +/- 666.67, N = 3 SE +/- 176.38, N = 3 SE +/- 633.33, N = 3 SE +/- 185.59, N = 3 329667 306667 224333 57433 56167 49567 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory ZFS mirror 8xNVME Pool ext4 Crucial P5 Plus 1TB NVME ZFS zraid1 8xNVME Pool no Compression ZFS zraid1 8xNVME Pool ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 400 800 1200 1600 2000 SE +/- 4.16, N = 3 SE +/- 4.91, N = 3 SE +/- 8.99, N = 3 SE +/- 1.76, N = 3 1783 1623 1337 1327 1225 476 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory ext4 Crucial P5 Plus 1TB NVME ext4 mdadm raid5 4xNVME ZFS mirror 8xNVME Pool ZFS zraid1 8xNVME Pool no Compression ZFS zraid1 8xNVME Pool ZFS zraid1 4xNVME Pool 50K 100K 150K 200K 250K SE +/- 666.67, N = 3 SE +/- 569.60, N = 3 SE +/- 88.19, N = 3 SE +/- 57.74, N = 3 SE +/- 33.33, N = 3 SE +/- 57.74, N = 3 224667 82967 68333 48200 45367 41500 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory ext4 mdadm raid5 4xNVME ZFS mirror 8xNVME Pool ZFS zraid1 4xNVME Pool ZFS zraid1 8xNVME Pool ZFS zraid1 8xNVME Pool no Compression ext4 Crucial P5 Plus 1TB NVME 1400 2800 4200 5600 7000 SE +/- 3.61, N = 3 SE +/- 18.84, N = 3 SE +/- 18.75, N = 3 SE +/- 25.32, N = 3 SE +/- 23.68, N = 3 6706 2073 2034 1999 1995 1765 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory ext4 Crucial P5 Plus 1TB NVME ZFS mirror 8xNVME Pool ZFS zraid1 8xNVME Pool no Compression ZFS zraid1 4xNVME Pool ZFS zraid1 8xNVME Pool ext4 mdadm raid5 4xNVME 70K 140K 210K 280K 350K SE +/- 4666.67, N = 3 SE +/- 2516.61, N = 3 SE +/- 1000.00, N = 3 SE +/- 1855.92, N = 3 SE +/- 3382.96, N = 3 SE +/- 2728.45, N = 3 335667 332000 331000 329333 326333 221333 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory ZFS mirror 8xNVME Pool ext4 Crucial P5 Plus 1TB NVME ZFS zraid1 8xNVME Pool no Compression ZFS zraid1 8xNVME Pool ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 400 800 1200 1600 2000 SE +/- 6.01, N = 3 SE +/- 2.85, N = 3 SE +/- 11.98, N = 15 SE +/- 2.33, N = 3 SE +/- 2.91, N = 3 SE +/- 0.88, N = 3 1774 1614 1220 1186 1029 650 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory ext4 Crucial P5 Plus 1TB NVME ZFS mirror 8xNVME Pool ZFS zraid1 8xNVME Pool no Compression ZFS zraid1 4xNVME Pool ZFS zraid1 8xNVME Pool ext4 mdadm raid5 4xNVME 50K 100K 150K 200K 250K SE +/- 1000.00, N = 3 SE +/- 333.33, N = 3 SE +/- 333.33, N = 3 SE +/- 666.67, N = 3 SE +/- 333.33, N = 3 SE +/- 577.35, N = 3 228000 197333 174667 171333 170667 115000 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory ZFS mirror 8xNVME Pool ZFS zraid1 4xNVME Pool ZFS zraid1 8xNVME Pool no Compression ZFS zraid1 8xNVME Pool ext4 Crucial P5 Plus 1TB NVME 1400 2800 4200 5600 7000 SE +/- 23.92, N = 3 SE +/- 58.64, N = 3 SE +/- 41.66, N = 3 SE +/- 13.37, N = 3 6617 4245 4170 4037 3538 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory ext4 Crucial P5 Plus 1TB NVME ext4 mdadm raid5 4xNVME ZFS mirror 8xNVME Pool ZFS zraid1 8xNVME Pool ZFS zraid1 8xNVME Pool no Compression ZFS zraid1 4xNVME Pool 300 600 900 1200 1500 SE +/- 3.48, N = 3 SE +/- 5.17, N = 3 SE +/- 1.86, N = 3 SE +/- 0.67, N = 3 SE +/- 2.67, N = 3 SE +/- 0.88, N = 3 1286 1199 877 224 219 194 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory ZFS mirror 8xNVME Pool ext4 Crucial P5 Plus 1TB NVME ZFS zraid1 8xNVME Pool no Compression ZFS zraid1 8xNVME Pool ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 800 1600 2400 3200 4000 SE +/- 8.25, N = 3 SE +/- 10.27, N = 3 SE +/- 6.39, N = 3 SE +/- 17.98, N = 3 SE +/- 3.53, N = 3 3573 3252 2681 2661 2458 958 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory ext4 Crucial P5 Plus 1TB NVME ext4 mdadm raid5 4xNVME ZFS mirror 8xNVME Pool ZFS zraid1 8xNVME Pool no Compression ZFS zraid1 8xNVME Pool ZFS zraid1 4xNVME Pool 200 400 600 800 1000 SE +/- 2.67, N = 3 SE +/- 2.00, N = 3 878 324 267 188 177 162 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory ZFS mirror 8xNVME Pool ZFS zraid1 4xNVME Pool ZFS zraid1 8xNVME Pool ZFS zraid1 8xNVME Pool no Compression ext4 Crucial P5 Plus 1TB NVME 900 1800 2700 3600 4500 SE +/- 37.36, N = 3 SE +/- 37.36, N = 3 SE +/- 50.44, N = 3 SE +/- 47.58, N = 3 4154 4075 4005 3998 3538 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory ext4 Crucial P5 Plus 1TB NVME ZFS mirror 8xNVME Pool ZFS zraid1 8xNVME Pool no Compression ZFS zraid1 4xNVME Pool ZFS zraid1 8xNVME Pool ext4 mdadm raid5 4xNVME 300 600 900 1200 1500 SE +/- 17.89, N = 3 SE +/- 9.94, N = 3 SE +/- 4.84, N = 3 SE +/- 7.09, N = 3 SE +/- 13.25, N = 3 SE +/- 11.46, N = 3 1311 1296 1294 1287 1275 865 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory ZFS mirror 8xNVME Pool ext4 Crucial P5 Plus 1TB NVME ZFS zraid1 8xNVME Pool no Compression ZFS zraid1 8xNVME Pool ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 800 1600 2400 3200 4000 SE +/- 11.93, N = 3 SE +/- 5.70, N = 3 SE +/- 23.91, N = 15 SE +/- 4.67, N = 3 SE +/- 6.36, N = 3 SE +/- 1.73, N = 3 3555 3236 2446 2381 2065 1308 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory ext4 Crucial P5 Plus 1TB NVME ZFS mirror 8xNVME Pool ZFS zraid1 8xNVME Pool no Compression ZFS zraid1 4xNVME Pool ZFS zraid1 8xNVME Pool ext4 mdadm raid5 4xNVME 200 400 600 800 1000 SE +/- 3.93, N = 3 SE +/- 0.67, N = 3 SE +/- 2.08, N = 3 SE +/- 1.86, N = 3 SE +/- 1.67, N = 3 SE +/- 1.15, N = 3 891 771 682 670 665 449 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Dbench 12 Clients OpenBenchmarking.org MB/s, More Is Better Dbench 4.0 12 Clients ZFS zraid1 8xNVME Pool no Compression ZFS mirror 8xNVME Pool ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME ZFS zraid1 8xNVME Pool ext4 Crucial P5 Plus 1TB NVME 600 1200 1800 2400 3000 SE +/- 16.46, N = 3 SE +/- 3.38, N = 3 SE +/- 5.07, N = 3 SE +/- 21.52, N = 3 SE +/- 2.03, N = 3 SE +/- 1.20, N = 3 2783.64 2730.75 2626.03 2484.85 2472.42 658.79 1. (CC) gcc options: -lpopt -O2
Dbench 1 Clients OpenBenchmarking.org MB/s, More Is Better Dbench 4.0 1 Clients ZFS zraid1 8xNVME Pool no Compression ext4 mdadm raid5 4xNVME ZFS mirror 8xNVME Pool ZFS zraid1 4xNVME Pool ZFS zraid1 8xNVME Pool ext4 Crucial P5 Plus 1TB NVME 100 200 300 400 500 SE +/- 1.41, N = 3 SE +/- 0.61, N = 3 SE +/- 0.70, N = 3 SE +/- 0.85, N = 3 SE +/- 0.25, N = 3 SE +/- 0.03, N = 3 458.91 453.17 435.70 410.17 380.35 102.99 1. (CC) gcc options: -lpopt -O2
Compile Bench Test: Compile OpenBenchmarking.org MB/s, More Is Better Compile Bench 0.6 Test: Compile ext4 Crucial P5 Plus 1TB NVME ext4 mdadm raid5 4xNVME ZFS mirror 8xNVME Pool ZFS zraid1 4xNVME Pool ZFS zraid1 8xNVME Pool ZFS zraid1 8xNVME Pool no Compression 300 600 900 1200 1500 SE +/- 0.00, N = 3 SE +/- 11.73, N = 3 SE +/- 9.96, N = 3 SE +/- 6.06, N = 3 SE +/- 7.47, N = 3 SE +/- 3.89, N = 3 1483.07 1478.00 1408.42 1398.57 1390.71 1247.36
Compile Bench Test: Initial Create OpenBenchmarking.org MB/s, More Is Better Compile Bench 0.6 Test: Initial Create ext4 Crucial P5 Plus 1TB NVME ext4 mdadm raid5 4xNVME ZFS zraid1 4xNVME Pool ZFS zraid1 8xNVME Pool ZFS mirror 8xNVME Pool ZFS zraid1 8xNVME Pool no Compression 90 180 270 360 450 SE +/- 1.23, N = 3 SE +/- 0.77, N = 3 SE +/- 1.57, N = 3 SE +/- 0.50, N = 3 SE +/- 1.18, N = 3 SE +/- 1.46, N = 3 423.67 411.07 222.04 221.83 216.21 206.77
Compile Bench Test: Read Compiled Tree OpenBenchmarking.org MB/s, More Is Better Compile Bench 0.6 Test: Read Compiled Tree ext4 Crucial P5 Plus 1TB NVME ext4 mdadm raid5 4xNVME ZFS zraid1 8xNVME Pool no Compression ZFS mirror 8xNVME Pool ZFS zraid1 8xNVME Pool ZFS zraid1 4xNVME Pool 600 1200 1800 2400 3000 SE +/- 8.94, N = 3 SE +/- 42.24, N = 3 SE +/- 19.98, N = 3 SE +/- 12.22, N = 3 SE +/- 8.89, N = 3 SE +/- 8.00, N = 3 2747.01 2642.03 1694.75 1454.31 1217.80 1212.73
PostMark Disk Transaction Performance OpenBenchmarking.org TPS, More Is Better PostMark 1.51 Disk Transaction Performance ext4 Crucial P5 Plus 1TB NVME ext4 mdadm raid5 4xNVME ZFS mirror 8xNVME Pool ZFS zraid1 4xNVME Pool ZFS zraid1 8xNVME Pool no Compression ZFS zraid1 8xNVME Pool 1100 2200 3300 4400 5500 SE +/- 35.33, N = 3 SE +/- 34.00, N = 3 SE +/- 14.67, N = 3 SE +/- 14.33, N = 3 SE +/- 29.00, N = 3 5137 5068 3318 3289 3275 3275 1. (CC) gcc options: -O3
SQLite Threads / Copies: 1 OpenBenchmarking.org Seconds, Fewer Is Better SQLite 3.30.1 Threads / Copies: 1 ZFS mirror 8xNVME Pool ZFS zraid1 4xNVME Pool ext4 Crucial P5 Plus 1TB NVME ZFS zraid1 8xNVME Pool no Compression ZFS zraid1 8xNVME Pool ext4 mdadm raid5 4xNVME 2 4 6 8 10 SE +/- 0.055, N = 3 SE +/- 0.071, N = 7 SE +/- 0.067, N = 3 SE +/- 0.104, N = 3 SE +/- 0.096, N = 3 SE +/- 0.069, N = 10 7.252 7.852 8.367 8.379 8.380 8.810 1. (CC) gcc options: -O2 -lz -lm -ldl -lpthread
SQLite Threads / Copies: 8 OpenBenchmarking.org Seconds, Fewer Is Better SQLite 3.30.1 Threads / Copies: 8 ZFS mirror 8xNVME Pool ZFS zraid1 4xNVME Pool ZFS zraid1 8xNVME Pool no Compression ZFS zraid1 8xNVME Pool ext4 Crucial P5 Plus 1TB NVME ext4 mdadm raid5 4xNVME 5 10 15 20 25 SE +/- 0.057, N = 3 SE +/- 0.079, N = 3 SE +/- 0.170, N = 3 SE +/- 0.174, N = 3 SE +/- 0.009, N = 3 SE +/- 0.047, N = 3 9.043 11.267 12.827 12.857 15.767 20.005 1. (CC) gcc options: -O2 -lz -lm -ldl -lpthread
SQLite Threads / Copies: 32 OpenBenchmarking.org Seconds, Fewer Is Better SQLite 3.30.1 Threads / Copies: 32 ZFS mirror 8xNVME Pool ZFS zraid1 4xNVME Pool ZFS zraid1 8xNVME Pool no Compression ZFS zraid1 8xNVME Pool ext4 Crucial P5 Plus 1TB NVME ext4 mdadm raid5 4xNVME 7 14 21 28 35 SE +/- 0.05, N = 3 SE +/- 0.08, N = 3 SE +/- 0.07, N = 3 SE +/- 0.04, N = 3 SE +/- 0.00, N = 3 SE +/- 0.01, N = 3 17.74 23.19 26.89 27.00 27.58 31.79 1. (CC) gcc options: -O2 -lz -lm -ldl -lpthread
SQLite Threads / Copies: 64 OpenBenchmarking.org Seconds, Fewer Is Better SQLite 3.30.1 Threads / Copies: 64 ZFS mirror 8xNVME Pool ZFS zraid1 4xNVME Pool ZFS zraid1 8xNVME Pool no Compression ZFS zraid1 8xNVME Pool ext4 Crucial P5 Plus 1TB NVME ext4 mdadm raid5 4xNVME 10 20 30 40 50 SE +/- 0.08, N = 3 SE +/- 0.01, N = 3 SE +/- 0.01, N = 3 SE +/- 0.05, N = 3 SE +/- 0.01, N = 3 SE +/- 0.13, N = 3 36.60 37.45 40.00 40.30 40.98 45.72 1. (CC) gcc options: -O2 -lz -lm -ldl -lpthread
SQLite Threads / Copies: 128 OpenBenchmarking.org Seconds, Fewer Is Better SQLite 3.30.1 Threads / Copies: 128 ext4 Crucial P5 Plus 1TB NVME ZFS zraid1 4xNVME Pool ZFS zraid1 8xNVME Pool no Compression ZFS mirror 8xNVME Pool ZFS zraid1 8xNVME Pool ext4 mdadm raid5 4xNVME 20 40 60 80 100 SE +/- 0.03, N = 3 SE +/- 0.10, N = 3 SE +/- 0.27, N = 3 SE +/- 0.16, N = 3 SE +/- 0.10, N = 3 SE +/- 0.28, N = 3 59.89 76.09 77.05 77.88 78.15 80.15 1. (CC) gcc options: -O2 -lz -lm -ldl -lpthread
Phoronix Test Suite v10.8.5