pts-disk-different-nvmes AMD Ryzen Threadripper 1900X 8-Core testing with a Gigabyte X399 DESIGNARE EX-CF (F13a BIOS) and NVIDIA Quadro P400 on Debian 11 via the Phoronix Test Suite.
HTML result view exported from: https://openbenchmarking.org/result/2211127-NE-PTSDISKDI57&sor&grs .
pts-disk-different-nvmes Processor Motherboard Chipset Memory Disk Graphics Audio Monitor Network OS Kernel Compiler File-System Screen Resolution ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME AMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads) Gigabyte X399 DESIGNARE EX-CF (F13a BIOS) AMD 17h 64GB Samsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8 NVIDIA Quadro P400 NVIDIA GP107GL HD Audio DELL S2340T 4 x Intel I350 + Intel 8265 / 8275 Debian 11 5.10.0-19-amd64 (x86_64) GCC 10.2.1 20210110 zfs 1920x1080 ext4 OpenBenchmarking.org Kernel Details - Transparent Huge Pages: always Compiler Details - --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-mutex --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-10-Km9U7s/gcc-10-10.2.1/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-10-Km9U7s/gcc-10-10.2.1/debian/tmp-gcn/usr,hsa --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details - Scaling Governor: acpi-cpufreq schedutil (Boost: Enabled) - CPU Microcode: 0x8001137 Disk Scheduler Details - ZFS zraid1 4xNVME Pool: NONE Python Details - Python 3.9.2 Security Details - itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Mitigation of untrained return thunk; SMT vulnerable + spec_store_bypass: Mitigation of SSB disabled via prctl and seccomp + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional STIBP: disabled RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected Disk Details - ext4 mdadm raid5 4xNVME: NONE / relatime,rw,stripe=384 / raid5 nvme4n1p1[4] nvme3n1p1[2] nvme2n1p1[1] nvme1n1p1[0] Block Size: 4096
pts-disk-different-nvmes fio: Rand Read - Linux AIO - No - Yes - 4KB - Default Test Directory fio: Rand Read - Linux AIO - No - Yes - 4KB - Default Test Directory fio: Seq Read - Linux AIO - No - Yes - 2MB - Default Test Directory fio: Rand Read - Linux AIO - No - Yes - 2MB - Default Test Directory fs-mark: 5000 Files, 1MB Size, 4 Threads fio: Rand Write - Linux AIO - No - Yes - 2MB - Default Test Directory fio: Rand Write - Linux AIO - No - Yes - 2MB - Default Test Directory fs-mark: 4000 Files, 32 Sub Dirs, 1MB Size compilebench: Read Compiled Tree fio: Rand Write - Linux AIO - No - Yes - 4KB - Default Test Directory fio: Rand Write - Linux AIO - No - Yes - 4KB - Default Test Directory compilebench: Initial Create sqlite: 8 fio: Seq Write - Linux AIO - No - Yes - 2MB - Default Test Directory fio: Seq Write - Linux AIO - No - Yes - 2MB - Default Test Directory postmark: Disk Transaction Performance fio: Seq Write - Linux AIO - No - Yes - 4KB - Default Test Directory fio: Seq Write - Linux AIO - No - Yes - 4KB - Default Test Directory fio: Seq Read - Linux AIO - No - Yes - 4KB - Default Test Directory fio: Seq Read - Linux AIO - No - Yes - 4KB - Default Test Directory sqlite: 32 fs-mark: 1000 Files, 1MB Size, No Sync/FSync sqlite: 64 sqlite: 1 dbench: 1 Clients dbench: 12 Clients compilebench: Compile sqlite: 128 fio: Seq Read - Linux AIO - No - Yes - 2MB - Default Test Directory fio: Rand Read - Linux AIO - No - Yes - 2MB - Default Test Directory fs-mark: 1000 Files, 1MB Size ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 49567 194 2034 2119 1632.7 1225 2458 684.2 1212.73 162 41500 222.04 11.267 1029 2065 3289 670 171333 329333 1287 23.191 1359.3 37.448 7.852 410.171 2626.03 1398.57 76.088 4075 4245 646.5 306667 1199 6706 6666 537.9 476 958 284.7 2642.03 324 82967 411.07 20.005 650 1308 5068 449 115000 221333 865 31.793 1733.3 45.720 8.810 453.170 2484.85 1478.00 80.145 254.0 OpenBenchmarking.org
Flexible IO Tester Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory ext4 mdadm raid5 4xNVME ZFS zraid1 4xNVME Pool 70K 140K 210K 280K 350K SE +/- 1333.33, N = 3 SE +/- 185.59, N = 3 306667 49567 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory ext4 mdadm raid5 4xNVME ZFS zraid1 4xNVME Pool 300 600 900 1200 1500 SE +/- 5.17, N = 3 SE +/- 0.88, N = 3 1199 194 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory ext4 mdadm raid5 4xNVME ZFS zraid1 4xNVME Pool 1400 2800 4200 5600 7000 SE +/- 3.61, N = 3 SE +/- 18.75, N = 3 6706 2034 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory ext4 mdadm raid5 4xNVME ZFS zraid1 4xNVME Pool 1400 2800 4200 5600 7000 SE +/- 14.99, N = 3 SE +/- 29.49, N = 3 6666 2119 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
FS-Mark Test: 5000 Files, 1MB Size, 4 Threads OpenBenchmarking.org Files/s, More Is Better FS-Mark 3.3 Test: 5000 Files, 1MB Size, 4 Threads ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 400 800 1200 1600 2000 SE +/- 4.95, N = 3 SE +/- 4.55, N = 12 1632.7 537.9
Flexible IO Tester Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 300 600 900 1200 1500 SE +/- 8.99, N = 3 SE +/- 1.76, N = 3 1225 476 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 500 1000 1500 2000 2500 SE +/- 17.98, N = 3 SE +/- 3.53, N = 3 2458 958 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
FS-Mark Test: 4000 Files, 32 Sub Dirs, 1MB Size OpenBenchmarking.org Files/s, More Is Better FS-Mark 3.3 Test: 4000 Files, 32 Sub Dirs, 1MB Size ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 150 300 450 600 750 SE +/- 7.90, N = 3 SE +/- 2.79, N = 3 684.2 284.7
Compile Bench Test: Read Compiled Tree OpenBenchmarking.org MB/s, More Is Better Compile Bench 0.6 Test: Read Compiled Tree ext4 mdadm raid5 4xNVME ZFS zraid1 4xNVME Pool 600 1200 1800 2400 3000 SE +/- 42.24, N = 3 SE +/- 8.00, N = 3 2642.03 1212.73
Flexible IO Tester Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory ext4 mdadm raid5 4xNVME ZFS zraid1 4xNVME Pool 70 140 210 280 350 SE +/- 2.00, N = 3 324 162 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.29 Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory ext4 mdadm raid5 4xNVME ZFS zraid1 4xNVME Pool 20K 40K 60K 80K 100K SE +/- 569.60, N = 3 SE +/- 57.74, N = 3 82967 41500 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Compile Bench Test: Initial Create OpenBenchmarking.org MB/s, More Is Better Compile Bench 0.6 Test: Initial Create ext4 mdadm raid5 4xNVME ZFS zraid1 4xNVME Pool 90 180 270 360 450 SE +/- 0.77, N = 3 SE +/- 1.57, N = 3 411.07 222.04
SQLite Threads / Copies: 8 OpenBenchmarking.org Seconds, Fewer Is Better SQLite 3.30.1 Threads / Copies: 8 ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 5 10 15 20 25 SE +/- 0.08, N = 3 SE +/- 0.05, N = 3 11.27 20.01 1. (CC) gcc options: -O2 -lz -lm -ldl -lpthread
Flexible IO Tester Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 200 400 600 800 1000 SE +/- 2.91, N = 3 SE +/- 0.88, N = 3 1029 650 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 400 800 1200 1600 2000 SE +/- 6.36, N = 3 SE +/- 1.73, N = 3 2065 1308 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
PostMark Disk Transaction Performance OpenBenchmarking.org TPS, More Is Better PostMark 1.51 Disk Transaction Performance ext4 mdadm raid5 4xNVME ZFS zraid1 4xNVME Pool 1100 2200 3300 4400 5500 SE +/- 34.00, N = 3 5068 3289 1. (CC) gcc options: -O3
Flexible IO Tester Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 140 280 420 560 700 SE +/- 1.86, N = 3 SE +/- 1.15, N = 3 670 449 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.29 Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 40K 80K 120K 160K 200K SE +/- 666.67, N = 3 SE +/- 577.35, N = 3 171333 115000 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org IOPS, More Is Better Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 70K 140K 210K 280K 350K SE +/- 1855.92, N = 3 SE +/- 2728.45, N = 3 329333 221333 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 300 600 900 1200 1500 SE +/- 7.09, N = 3 SE +/- 11.46, N = 3 1287 865 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
SQLite Threads / Copies: 32 OpenBenchmarking.org Seconds, Fewer Is Better SQLite 3.30.1 Threads / Copies: 32 ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 7 14 21 28 35 SE +/- 0.08, N = 3 SE +/- 0.01, N = 3 23.19 31.79 1. (CC) gcc options: -O2 -lz -lm -ldl -lpthread
FS-Mark Test: 1000 Files, 1MB Size, No Sync/FSync OpenBenchmarking.org Files/s, More Is Better FS-Mark 3.3 Test: 1000 Files, 1MB Size, No Sync/FSync ext4 mdadm raid5 4xNVME ZFS zraid1 4xNVME Pool 400 800 1200 1600 2000 SE +/- 19.84, N = 4 SE +/- 12.01, N = 15 1733.3 1359.3
SQLite Threads / Copies: 64 OpenBenchmarking.org Seconds, Fewer Is Better SQLite 3.30.1 Threads / Copies: 64 ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 10 20 30 40 50 SE +/- 0.01, N = 3 SE +/- 0.13, N = 3 37.45 45.72 1. (CC) gcc options: -O2 -lz -lm -ldl -lpthread
SQLite Threads / Copies: 1 OpenBenchmarking.org Seconds, Fewer Is Better SQLite 3.30.1 Threads / Copies: 1 ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 2 4 6 8 10 SE +/- 0.071, N = 7 SE +/- 0.069, N = 10 7.852 8.810 1. (CC) gcc options: -O2 -lz -lm -ldl -lpthread
Dbench 1 Clients OpenBenchmarking.org MB/s, More Is Better Dbench 4.0 1 Clients ext4 mdadm raid5 4xNVME ZFS zraid1 4xNVME Pool 100 200 300 400 500 SE +/- 0.61, N = 3 SE +/- 0.85, N = 3 453.17 410.17 1. (CC) gcc options: -lpopt -O2
Dbench 12 Clients OpenBenchmarking.org MB/s, More Is Better Dbench 4.0 12 Clients ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 600 1200 1800 2400 3000 SE +/- 5.07, N = 3 SE +/- 21.52, N = 3 2626.03 2484.85 1. (CC) gcc options: -lpopt -O2
Compile Bench Test: Compile OpenBenchmarking.org MB/s, More Is Better Compile Bench 0.6 Test: Compile ext4 mdadm raid5 4xNVME ZFS zraid1 4xNVME Pool 300 600 900 1200 1500 SE +/- 11.73, N = 3 SE +/- 6.06, N = 3 1478.00 1398.57
SQLite Threads / Copies: 128 OpenBenchmarking.org Seconds, Fewer Is Better SQLite 3.30.1 Threads / Copies: 128 ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 20 40 60 80 100 SE +/- 0.10, N = 3 SE +/- 0.28, N = 3 76.09 80.15 1. (CC) gcc options: -O2 -lz -lm -ldl -lpthread
Flexible IO Tester Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.29 Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory ZFS zraid1 4xNVME Pool 900 1800 2700 3600 4500 SE +/- 37.36, N = 3 4075 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
Flexible IO Tester Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory OpenBenchmarking.org MB/s, More Is Better Flexible IO Tester 3.29 Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory ZFS zraid1 4xNVME Pool 900 1800 2700 3600 4500 SE +/- 58.64, N = 3 4245 1. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native
FS-Mark Test: 1000 Files, 1MB Size OpenBenchmarking.org Files/s, More Is Better FS-Mark 3.3 Test: 1000 Files, 1MB Size ZFS zraid1 4xNVME Pool ext4 mdadm raid5 4xNVME 140 280 420 560 700 SE +/- 5.49, N = 15 SE +/- 9.50, N = 15 646.5 254.0
Phoronix Test Suite v10.8.5