storagetest_again balde nomem raid6: Processor: Intel Core i7-3770 @ 3.90GHz (4 Cores / 8 Threads), Motherboard: ASUS P8H77-M PRO (9012 BIOS), Chipset: Intel Xeon E3-1200 v2/3rd, Memory: 32GB, Disk: 7 x 4001GB HGST HMS5C4040BL + 256GB Generic, Graphics: ASUS Intel Xeon E3-1200 v2/3rd Gen Core (1150MHz), Network: Realtek RTL8111/8168/8411 OS: Debian 11, Kernel: 5.10.120+truenas (x86_64), Vulkan: 1.0.2, Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1024x768 mem raid6: Processor: Intel Core i7-3770 @ 3.90GHz (4 Cores / 8 Threads), Motherboard: ASUS P8H77-M PRO (9012 BIOS), Chipset: Intel Xeon E3-1200 v2/3rd, Memory: 32GB, Disk: 7 x 4001GB HGST HMS5C4040BL + 256GB Generic, Graphics: ASUS Intel Xeon E3-1200 v2/3rd Gen Core (1150MHz), Network: Realtek RTL8111/8168/8411 OS: Debian 11, Kernel: 5.10.120+truenas (x86_64), Vulkan: 1.0.2, Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1024x768 nomem mirror: Processor: Intel Core i7-3770 @ 3.90GHz (4 Cores / 8 Threads), Motherboard: ASUS P8H77-M PRO (9012 BIOS), Chipset: Intel Xeon E3-1200 v2/3rd, Memory: 32GB, Disk: 7 x 4001GB HGST HMS5C4040BL + 256GB Generic, Graphics: ASUS Intel Xeon E3-1200 v2/3rd Gen Core (1150MHz), Network: Realtek RTL8111/8168/8411 OS: Debian 11, Kernel: 5.10.120+truenas (x86_64), Vulkan: 1.0.2, Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1024x768 mem mirror: Processor: Intel Core i7-3770 @ 3.90GHz (4 Cores / 8 Threads), Motherboard: ASUS P8H77-M PRO (9012 BIOS), Chipset: Intel Xeon E3-1200 v2/3rd, Memory: 32GB, Disk: 7 x 4001GB HGST HMS5C4040BL + 256GB Generic, Graphics: ASUS Intel Xeon E3-1200 v2/3rd Gen Core (1150MHz), Network: Realtek RTL8111/8168/8411 OS: Debian 11, Kernel: 5.10.120+truenas (x86_64), Vulkan: 1.0.2, Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1024x768 mem raid6 aes-gcm-256: Processor: Intel Core i7-3770 @ 3.90GHz (4 Cores / 8 Threads), Motherboard: ASUS P8H77-M PRO (9012 BIOS), Chipset: Intel Xeon E3-1200 v2/3rd, Memory: 32GB, Disk: 7 x 4001GB HGST HMS5C4040BL + 256GB Generic, Graphics: ASUS Intel Xeon E3-1200 v2/3rd Gen Core (1150MHz), Network: Realtek RTL8111/8168/8411 OS: Debian 11, Kernel: 5.10.120+truenas (x86_64), Vulkan: 1.0.2, Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1024x768 mem raid6 aes-256-CCM: Processor: Intel Core i7-3770 @ 3.90GHz (4 Cores / 8 Threads), Motherboard: ASUS P8H77-M PRO (9012 BIOS), Chipset: Intel Xeon E3-1200 v2/3rd, Memory: 32GB, Disk: 7 x 4001GB HGST HMS5C4040BL + 256GB Generic, Graphics: ASUS Intel Xeon E3-1200 v2/3rd Gen Core (1150MHz), Network: Realtek RTL8111/8168/8411 OS: Debian 11, Kernel: 5.10.120+truenas (x86_64), Vulkan: 1.0.2, Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 1024x768 susy: Processor: AMD FX-4170 @ 4.20GHz (2 Cores / 4 Threads), Motherboard: Gigabyte GA-990FXA-D3 (F3 BIOS), Chipset: AMD RD9x0/RX980, Memory: 16GB, Disk: 5 x 6001GB Western Digital WD60EFZX-68B + 120GB Corsair Force GT + 6001GB HDWN160, Graphics: Gigabyte NVIDIA GeForce GTX 1050 Ti 4GB, Audio: Realtek ALC889, Network: Realtek RTL8111/8168/8411 OS: Debian 11, Kernel: 5.10.120+truenas (x86_64), Display Driver: NVIDIA, Vulkan: 1.0.2, Compiler: GCC 10.2.1 20210110, File-System: zfs, Screen Resolution: 800x600 blade: Processor: Intel Core i7-10875H @ 5.10GHz (8 Cores / 16 Threads), Motherboard: Razer DA760 v4 (1.03 BIOS), Chipset: Intel Comet Lake PCH, Memory: 32GB, Disk: 1024GB SAMSUNG MZVLB1T0HBLR-00A00 + 1024GB INTEL SSDPEKNW010T8, Graphics: NVIDIA GeForce RTX 3080 Laptop GPU 8GB, Audio: Realtek ALC298, Network: Realtek RTL8125 2.5GbE + Intel Wi-Fi 6 AX210/AX211/AX411 OS: ManjaroLinux 21.3.6, Kernel: 5.15.59-1-MANJARO (x86_64), Desktop: Xfce 4.16, Display Server: X Server 1.21.1.4, Display Driver: NVIDIA 515.65.01, OpenGL: 4.6.0 NVIDIA 515.65.01, Compiler: GCC 12.1.1 20220730 + Clang 14.0.6 + LLVM 14.0.6 + CUDA 11.7, File-System: zfs, Screen Resolution: 5120x3240 blade2: Processor: Intel Core i7-10875H @ 5.10GHz (8 Cores / 16 Threads), Motherboard: Razer DA760 v4 (1.03 BIOS), Chipset: Intel Comet Lake PCH, Memory: 32GB, Disk: 1024GB SAMSUNG MZVLB1T0HBLR-00A00 + 1024GB INTEL SSDPEKNW010T8, Graphics: NVIDIA GeForce RTX 3080 Laptop GPU 8GB, Audio: Realtek ALC298, Network: Realtek RTL8125 2.5GbE + Intel Wi-Fi 6 AX210/AX211/AX411 OS: ManjaroLinux 21.3.6, Kernel: 5.15.59-1-MANJARO (x86_64), Desktop: Xfce 4.16, Display Server: X Server 1.21.1.4, Display Driver: NVIDIA 515.65.01, OpenGL: 4.6.0 NVIDIA 515.65.01, Compiler: GCC 12.1.1 20220730 + Clang 14.0.6 + LLVM 14.0.6 + CUDA 11.7, File-System: zfs, Screen Resolution: 5120x3240 SQLite 3.30.1 Threads / Copies: 32 Seconds < Lower Is Better nomem raid6 ........... 708.76 |=============================================== mem raid6 ............. 699.27 |============================================== nomem mirror .......... 30.99 |== mem mirror ............ 32.09 |== mem raid6 aes-gcm-256 . 36.76 |== mem raid6 aes-256-CCM . 40.06 |=== susy .................. 124.53 |======== blade ................. 36.60 |== blade2 ................ 23.75 |== Flexible IO Tester 3.29 Type: Random Read - Engine: Sync - Buffered: Yes - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory MB/s > Higher Is Better nomem raid6 ........... 215 |= mem raid6 ............. 5346 |================================== nomem mirror .......... 174 |= mem mirror ............ 5384 |================================== mem raid6 aes-gcm-256 . 5394 |================================== mem raid6 aes-256-CCM . 2635 |================= mem raid6 aes-256-CCM . 2595 |================ susy .................. 2224 |============== blade2 ................ 7727 |================================================= Flexible IO Tester 3.29 Type: Random Read - Engine: Sync - Buffered: Yes - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory IOPS > Higher Is Better nomem raid6 ........... 213 |= mem raid6 ............. 5350 |==================================== nomem mirror .......... 155 |= mem mirror ............ 5379 |==================================== mem raid6 aes-gcm-256 . 5393 |==================================== mem raid6 aes-256-CCM . 2611 |================== susy .................. 2199 |=============== blade2 ................ 7253 |================================================= Flexible IO Tester 3.29 Type: Random Read - Engine: Sync - Buffered: Yes - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better nomem raid6 ........... 3.454 | mem raid6 ............. 201.000 |=================== nomem mirror .......... 5.456 |= mem mirror ............ 197.000 |=================== mem raid6 aes-gcm-256 . 205.000 |==================== mem raid6 aes-256-CCM . 105.000 |========== susy .................. 116.000 |=========== blade2 ................ 481.000 |============================================== Flexible IO Tester 3.29 Type: Random Read - Engine: Sync - Buffered: Yes - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better nomem raid6 ........... 863 | mem raid6 ............. 51667 |==================== nomem mirror .......... 1364 |= mem mirror ............ 50433 |=================== mem raid6 aes-gcm-256 . 52367 |==================== mem raid6 aes-256-CCM . 26867 |========== susy .................. 29767 |=========== blade2 ................ 123053 |=============================================== Flexible IO Tester 3.29 Type: Random Write - Engine: Sync - Buffered: Yes - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory MB/s > Higher Is Better nomem raid6 ........... 306 |== mem raid6 ............. 382 |== nomem mirror .......... 309 |== mem mirror ............ 378 |== mem raid6 aes-gcm-256 . 416 |=== mem raid6 aes-gcm-256 . 411 |== mem raid6 aes-256-CCM . 7509 |============================================= susy .................. 447 |=== blade2 ................ 8088 |================================================= Flexible IO Tester 3.29 Type: Random Write - Engine: Sync - Buffered: Yes - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory IOPS > Higher Is Better nomem raid6 ........... 300 |= mem raid6 ............. 384 |== nomem mirror .......... 308 |= mem mirror ............ 377 |== mem raid6 aes-gcm-256 . 411 |== mem raid6 aes-256-CCM . 7144 |================================ susy .................. 443 |== blade2 ................ 10617 |================================================ Flexible IO Tester 3.29 Type: Random Write - Engine: Sync - Buffered: Yes - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better nomem raid6 ........... 40.6 |== mem raid6 ............. 358.0 |================== nomem mirror .......... 52.1 |=== mem mirror ............ 265.0 |============= mem raid6 aes-gcm-256 . 309.0 |=============== mem raid6 aes-256-CCM . 52.6 |=== susy .................. 56.6 |=== blade2 ................ 959.0 |================================================ Flexible IO Tester 3.29 Type: Random Write - Engine: Sync - Buffered: Yes - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better nomem raid6 ........... 10405 |== mem raid6 ............. 91500 |================== nomem mirror .......... 13333 |=== mem mirror ............ 67900 |============= mem raid6 aes-gcm-256 . 79000 |=============== mem raid6 aes-256-CCM . 13467 |=== susy .................. 14500 |=== blade2 ................ 245500 |=============================================== Flexible IO Tester 3.29 Type: Sequential Read - Engine: Sync - Buffered: Yes - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory MB/s > Higher Is Better nomem raid6 ........... 462 |=== mem raid6 ............. 5403 |======================================== nomem mirror .......... 561 |==== mem mirror ............ 5435 |======================================== mem raid6 aes-gcm-256 . 5445 |========================================= mem raid6 aes-256-CCM . 2623 |==================== susy .................. 2235 |================= blade2 ................ 6587 |================================================= Flexible IO Tester 3.29 Type: Sequential Read - Engine: Sync - Buffered: Yes - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory IOPS > Higher Is Better nomem raid6 ........... 460 |=== mem raid6 ............. 5408 |======================================== nomem mirror .......... 581 |==== mem mirror ............ 5434 |======================================== mem raid6 aes-gcm-256 . 5429 |======================================== mem raid6 aes-256-CCM . 2630 |==================== susy .................. 2234 |================= blade2 ................ 6586 |================================================= Flexible IO Tester 3.29 Type: Sequential Read - Engine: Sync - Buffered: Yes - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better nomem raid6 ........... 443 |========== mem raid6 ............. 1292 |============================ nomem mirror .......... 536 |============ mem mirror ............ 1299 |============================ mem raid6 aes-gcm-256 . 1278 |============================ mem raid6 aes-256-CCM . 622 |============== susy .................. 684 |=============== blade2 ................ 2236 |================================================= Flexible IO Tester 3.29 Type: Sequential Read - Engine: Sync - Buffered: Yes - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better nomem raid6 ........... 113333 |========= mem raid6 ............. 330667 |=========================== nomem mirror .......... 137333 |=========== mem mirror ............ 332667 |=========================== mem raid6 aes-gcm-256 . 327000 |=========================== mem raid6 aes-256-CCM . 159333 |============= susy .................. 175000 |============== blade2 ................ 572333 |=============================================== Flexible IO Tester 3.29 Type: Sequential Write - Engine: Sync - Buffered: Yes - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory MB/s > Higher Is Better nomem raid6 ........... 304 |== mem raid6 ............. 361 |== nomem mirror .......... 308 |== mem mirror ............ 339 |== mem raid6 aes-gcm-256 . 370 |== mem raid6 aes-256-CCM . 7582 |======================================= susy .................. 381 |== blade2 ................ 9464 |================================================= Flexible IO Tester 3.29 Type: Sequential Write - Engine: Sync - Buffered: Yes - Direct: Yes - Block Size: 1MB - Disk Target: Default Test Directory IOPS > Higher Is Better nomem raid6 ........... 302 |= mem raid6 ............. 349 |== nomem mirror .......... 308 |= mem mirror ............ 338 |= mem raid6 aes-gcm-256 . 372 |== mem raid6 aes-256-CCM . 7581 |================================= susy .................. 381 |== blade2 ................ 11044 |================================================ Flexible IO Tester 3.29 Type: Sequential Write - Engine: Sync - Buffered: Yes - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better nomem raid6 ........... 408 |=============== mem raid6 ............. 628 |====================== nomem mirror .......... 509 |================== mem mirror ............ 691 |========================= mem raid6 aes-gcm-256 . 663 |======================== mem raid6 aes-256-CCM . 393 |============== susy .................. 337 |============ blade2 ................ 1376 |================================================= Flexible IO Tester 3.29 Type: Sequential Write - Engine: Sync - Buffered: Yes - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better nomem raid6 ........... 104167 |============== mem raid6 ............. 161000 |===================== nomem mirror .......... 130333 |================= mem mirror ............ 176667 |======================== mem raid6 aes-gcm-256 . 169667 |======================= mem raid6 aes-256-CCM . 100467 |============= susy .................. 86333 |============ blade2 ................ 352333 |=============================================== FS-Mark 3.3 Test: 5000 Files, 1MB Size, 4 Threads Files/s > Higher Is Better nomem raid6 ........... 83.3 |= mem raid6 ............. 79.4 |= nomem mirror .......... 1763.4 |============== mem mirror ............ 4234.4 |================================ mem raid6 aes-gcm-256 . 4328.6 |================================= mem raid6 aes-256-CCM . 4517.5 |=================================== susy .................. 1573.4 |============ blade2 ................ 6130.3 |=============================================== Dbench 4.0 Client Count: 12 MB/s > Higher Is Better nomem raid6 ........... 139.86 |==== mem raid6 ............. 138.82 |==== nomem mirror .......... 1099.95 |============================= mem mirror ............ 1084.44 |============================ mem raid6 aes-gcm-256 . 855.98 |====================== mem raid6 aes-256-CCM . 857.60 |====================== susy .................. 288.23 |======== blade2 ................ 1759.32 |============================================== Compile Bench 0.6 Test: Compile MB/s > Higher Is Better nomem raid6 ........... 529.38 |============== mem raid6 ............. 1466.06 |======================================== nomem mirror .......... 508.91 |============== mem mirror ............ 1698.25 |============================================== mem raid6 aes-gcm-256 . 1206.78 |================================= mem raid6 aes-256-CCM . 932.38 |========================= susy .................. 729.76 |==================== blade2 ................ 952.92 |========================== Compile Bench 0.6 Test: Initial Create MB/s > Higher Is Better nomem raid6 ........... 151.06 |============================ mem raid6 ............. 185.88 |================================== nomem mirror .......... 171.46 |=============================== mem mirror ............ 198.26 |==================================== mem raid6 aes-gcm-256 . 178.35 |================================= mem raid6 aes-256-CCM . 175.24 |================================ susy .................. 106.79 |==================== blade2 ................ 257.20 |=============================================== Compile Bench 0.6 Test: Read Compiled Tree MB/s > Higher Is Better nomem raid6 ........... 671.46 |===================== mem raid6 ............. 980.65 |=============================== nomem mirror .......... 471.94 |=============== mem mirror ............ 1016.19 |================================ mem raid6 aes-gcm-256 . 891.12 |============================ mem raid6 aes-256-CCM . 616.12 |=================== susy .................. 621.22 |=================== blade2 ................ 1477.28 |============================================== PostMark 1.51 Disk Transaction Performance TPS > Higher Is Better nomem raid6 ........... 2650 |============================================== mem raid6 ............. 2762 |=============================================== nomem mirror .......... 2492 |=========================================== mem mirror ............ 2841 |================================================= mem raid6 aes-gcm-256 . 2852 |================================================= mem raid6 aes-256-CCM . 2443 |========================================== susy .................. 1304 |====================== blade2 ................ 800 |==============