dbox 2 x Intel Xeon L5640 testing with a Supermicro X8DTH and Matrox s MGA G200eW WPCM450 on Ubuntu 16.04 via the Phoronix Test Suite. 3sas_zfs_opts: Processor: 2 x Intel Xeon L5640 @ 2.27GHz (24 Cores), Motherboard: Supermicro X8DTH, Chipset: Intel 5520 I/O + ICH10R, Memory: 80896MB, Disk: 500GB Western Digital WD5000ABYS-0 + 500GB Seagate ST9500420ASG + 2 x 1000GB Seagate ST1000LM014-1EJ1 + 4 x 4001GB TOSHIBA HDWQ140 + 1000GB Western Digital WD10EZEX-00B + 2 x 4001GB HGST HDN726040AL + 64GB Type-C + 8GB Cruzer Blade + 3 x 1000GB Seagate ST31000528AS + Samsung SSD 950 PRO 512GB, Graphics: Matrox s MGA G200eW WPCM450, Network: Intel 82576 Gigabit Connection + Mellanox MT27500 Family OS: Ubuntu 16.04, Kernel: 4.10.0-35-lowlatency (x86_64), Compiler: GCC 5.4.0 20160609, File-System: zfs, Screen Resolution: 1024x768 3sas_zfs_opts_all_tests: Processor: 2 x Intel Xeon L5640 @ 2.27GHz (24 Cores), Motherboard: Supermicro X8DTH, Chipset: Intel 5520 I/O + ICH10R, Memory: 80896MB, Disk: 500GB Western Digital WD5000ABYS-0 + 500GB Seagate ST9500420ASG + 2 x 1000GB Seagate ST1000LM014-1EJ1 + 4 x 4001GB TOSHIBA HDWQ140 + 1000GB Western Digital WD10EZEX-00B + 2 x 4001GB HGST HDN726040AL + 64GB Type-C + 8GB Cruzer Blade + 3 x 1000GB Seagate ST31000528AS + Samsung SSD 950 PRO 512GB, Graphics: Matrox s MGA G200eW WPCM450, Network: Intel 82576 Gigabit Connection + Mellanox MT27500 Family OS: Ubuntu 16.04, Kernel: 4.10.0-35-lowlatency (x86_64), Compiler: GCC 5.4.0 20160609, File-System: zfs, Screen Resolution: 1024x768 3sas_zfs_z2_txg_inflight_32: Processor: 2 x Intel Xeon L5640 @ 2.27GHz (24 Cores), Motherboard: Supermicro X8DTH, Chipset: Intel 5520 I/O + ICH10R, Memory: 80896MB, Disk: 500GB Western Digital WD5000ABYS-0 + 500GB Seagate ST9500420ASG + 2 x 1000GB Seagate ST1000LM014-1EJ1 + 4 x 4001GB TOSHIBA HDWQ140 + 1000GB Western Digital WD10EZEX-00B + 2 x 4001GB HGST HDN726040AL + 64GB Type-C + 8GB Cruzer Blade + 3 x 1000GB Seagate ST31000528AS + Samsung SSD 950 PRO 512GB, Graphics: Matrox s MGA G200eW WPCM450, Network: Intel 82576 Gigabit Connection + Mellanox MT27500 Family OS: Ubuntu 16.04, Kernel: 4.10.0-35-lowlatency (x86_64), Compiler: GCC 5.4.0 20160609, File-System: zfs, Screen Resolution: 1024x768 3sas_zfs_z2_txg_128_volthrds_64: Processor: 2 x Intel Xeon L5640 @ 2.27GHz (24 Cores), Motherboard: Supermicro X8DTH, Chipset: Intel 5520 I/O + ICH10R, Memory: 80896MB, Disk: 500GB Western Digital WD5000ABYS-0 + 500GB Seagate ST9500420ASG + 2 x 1000GB Seagate ST1000LM014-1EJ1 + 4 x 4001GB TOSHIBA HDWQ140 + 1000GB Western Digital WD10EZEX-00B + 2 x 4001GB HGST HDN726040AL + 64GB Type-C + 8GB Cruzer Blade + 3 x 1000GB Seagate ST31000528AS + Samsung SSD 950 PRO 512GB, Graphics: Matrox s MGA G200eW WPCM450, Network: Intel 82576 Gigabit Connection + Mellanox MT27500 Family OS: Ubuntu 16.04, Kernel: 4.10.0-35-lowlatency (x86_64), Compiler: GCC 5.4.0 20160609, File-System: zfs, Screen Resolution: 1024x768 1sas_exp_zfs_z2_txg_128_volthrds_64: Processor: 2 x Intel Xeon L5640 @ 2.27GHz (24 Cores), Motherboard: Supermicro X8DTH, Chipset: Intel 5520 I/O + ICH10R, Memory: 80896MB, Disk: 8GB Cruzer Blade + 64GB Type-C + 3 x 1000GB Seagate ST31000528AS + 2 x 1000GB Seagate ST1000LM014-1EJ1 + 4 x 4001GB TOSHIBA HDWQ140 + 500GB Western Digital WD5000ABYS-0 + 500GB Seagate ST9500420ASG + 1000GB Western Digital WD10EZEX-00B + 2 x 4001GB HGST HDN726040AL + Samsung SSD 950 PRO 512GB, Graphics: Matrox s MGA G200eW WPCM450, Network: Intel 82576 Gigabit Connection + Mellanox MT27500 Family OS: Ubuntu 16.04, Kernel: 4.10.0-35-lowlatency (x86_64), Compiler: GCC 5.4.0 20160609, File-System: zfs, Screen Resolution: 1024x768 1sas_exp_zfs_z2_prefetch_rw_2_other_opts: Processor: 2 x Intel Xeon L5640 @ 2.27GHz (24 Cores), Motherboard: Supermicro X8DTH, Chipset: Intel 5520 I/O + ICH10R, Memory: 80896MB, Disk: 8GB Cruzer Blade + 64GB Type-C + 3 x 1000GB Seagate ST31000528AS + 2 x 1000GB Seagate ST1000LM014-1EJ1 + 4 x 4001GB TOSHIBA HDWQ140 + 500GB Western Digital WD5000ABYS-0 + 500GB Seagate ST9500420ASG + 1000GB Western Digital WD10EZEX-00B + 2 x 4001GB HGST HDN726040AL + Samsung SSD 950 PRO 512GB, Graphics: Matrox s MGA G200eW WPCM450, Network: Intel 82576 Gigabit Connection + Mellanox MT27500 Family OS: Ubuntu 16.04, Kernel: 4.10.0-35-lowlatency (x86_64), Compiler: GCC 5.4.0 20160609, File-System: zfs, Screen Resolution: 1024x768 sync_off_1sas_zfs_z2_prefetch_rw_2_other: Processor: 2 x Intel Xeon L5640 @ 2.27GHz (24 Cores), Motherboard: Supermicro X8DTH, Chipset: Intel 5520 I/O + ICH10R, Memory: 80896MB, Disk: 8GB Cruzer Blade + 64GB Type-C + 3 x 1000GB Seagate ST31000528AS + 2 x 1000GB Seagate ST1000LM014-1EJ1 + 4 x 4001GB TOSHIBA HDWQ140 + 500GB Western Digital WD5000ABYS-0 + 500GB Seagate ST9500420ASG + 1000GB Western Digital WD10EZEX-00B + 2 x 4001GB HGST HDN726040AL + Samsung SSD 950 PRO 512GB, Graphics: Matrox s MGA G200eW WPCM450, Network: Intel 82576 Gigabit Connection + Mellanox MT27500 Family OS: Ubuntu 16.04, Kernel: 4.10.0-35-lowlatency (x86_64), Compiler: GCC 5.4.0 20160609, File-System: zfs, Screen Resolution: 1024x768 sync_dva_1sas_z2_128_inflight_64_thrds: Processor: 2 x Intel Xeon L5640 @ 2.27GHz (24 Cores), Motherboard: Supermicro X8DTH, Chipset: Intel 5520 I/O + ICH10R, Memory: 80896MB, Disk: 8GB Cruzer Blade + 64GB Type-C + 3 x 1000GB Seagate ST31000528AS + 2 x 1000GB Seagate ST1000LM014-1EJ1 + 4 x 4001GB TOSHIBA HDWQ140 + 500GB Western Digital WD5000ABYS-0 + 500GB Seagate ST9500420ASG + 1000GB Western Digital WD10EZEX-00B + 2 x 4001GB HGST HDN726040AL + Samsung SSD 950 PRO 512GB, Graphics: Matrox s MGA G200eW WPCM450, Network: Intel 82576 Gigabit Connection + Mellanox MT27500 Family OS: Ubuntu 16.04, Kernel: 4.10.0-35-lowlatency (x86_64), Compiler: GCC 5.4.0 20160609, File-System: zfs, Screen Resolution: 1024x768 1sas_all_opts: Processor: 2 x Intel Xeon L5640 @ 2.27GHz (24 Cores), Motherboard: Supermicro X8DTH, Chipset: Intel 5520 I/O + ICH10R, Memory: 80896MB, Disk: 8GB Cruzer Blade + 64GB Type-C + 3 x 1000GB Seagate ST31000528AS + 2 x 1000GB Seagate ST1000LM014-1EJ1 + 4 x 4001GB TOSHIBA HDWQ140 + 500GB Western Digital WD5000ABYS-0 + 500GB Seagate ST9500420ASG + 1000GB Western Digital WD10EZEX-00B + 2 x 4001GB HGST HDN726040AL + Samsung SSD 950 PRO 512GB, Graphics: Matrox s MGA G200eW WPCM450, Network: Intel 82576 Gigabit Connection + Mellanox MT27500 Family OS: Ubuntu 16.04, Kernel: 4.10.0-35-lowlatency (x86_64), Compiler: GCC 5.4.0 20160609, File-System: zfs, Screen Resolution: 1024x768 1sas_all_opts_run2: Processor: 2 x Intel Xeon L5640 @ 2.27GHz (24 Cores), Motherboard: Supermicro X8DTH, Chipset: Intel 5520 I/O + ICH10R, Memory: 80896MB, Disk: 8GB Cruzer Blade + 64GB Type-C + 3 x 1000GB Seagate ST31000528AS + 2 x 1000GB Seagate ST1000LM014-1EJ1 + 4 x 4001GB TOSHIBA HDWQ140 + 500GB Western Digital WD5000ABYS-0 + 500GB Seagate ST9500420ASG + 1000GB Western Digital WD10EZEX-00B + 2 x 4001GB HGST HDN726040AL + Samsung SSD 950 PRO 512GB, Graphics: Matrox s MGA G200eW WPCM450, Network: Intel 82576 Gigabit Connection + Mellanox MT27500 Family OS: Ubuntu 16.04, Kernel: 4.10.0-35-lowlatency (x86_64), Compiler: GCC 5.4.0 20160609, File-System: zfs, Screen Resolution: 1024x768 1sas_all_deadline: Processor: 2 x Intel Xeon L5640 @ 2.27GHz (24 Cores), Motherboard: Supermicro X8DTH, Chipset: Intel 5520 I/O + ICH10R, Memory: 80896MB, Disk: 8GB Cruzer Blade + 64GB Type-C + 3 x 1000GB Seagate ST31000528AS + 2 x 1000GB Seagate ST1000LM014-1EJ1 + 4 x 4001GB TOSHIBA HDWQ140 + 500GB Western Digital WD5000ABYS-0 + 500GB Seagate ST9500420ASG + 1000GB Western Digital WD10EZEX-00B + 2 x 4001GB HGST HDN726040AL + Samsung SSD 950 PRO 512GB, Graphics: Matrox s MGA G200eW WPCM450, Network: Intel 82576 Gigabit Connection + Mellanox MT27500 Family OS: Ubuntu 16.04, Kernel: 4.10.0-35-lowlatency (x86_64), Compiler: GCC 5.4.0 20160609, File-System: zfs, Screen Resolution: 1024x768 1sas_all_opts_deadline: Processor: 2 x Intel Xeon L5640 @ 2.27GHz (24 Cores), Motherboard: Supermicro X8DTH, Chipset: Intel 5520 I/O + ICH10R, Memory: 80896MB, Disk: 8GB Cruzer Blade + 64GB Type-C + 3 x 1000GB Seagate ST31000528AS + 2 x 1000GB Seagate ST1000LM014-1EJ1 + 4 x 4001GB TOSHIBA HDWQ140 + 500GB Western Digital WD5000ABYS-0 + 500GB Seagate ST9500420ASG + 1000GB Western Digital WD10EZEX-00B + 2 x 4001GB HGST HDN726040AL + Samsung SSD 950 PRO 512GB, Graphics: Matrox s MGA G200eW WPCM450, Network: Intel 82576 Gigabit Connection + Mellanox MT27500 Family OS: Ubuntu 16.04, Kernel: 4.10.0-35-lowlatency (x86_64), Compiler: GCC 5.4.0 20160609, File-System: zfs, Screen Resolution: 1024x768 IOzone 3.465 Record Size: 1MB - File Size: 2GB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts ............................ 2801.53 |=========================== 3sas_zfs_opts_all_tests .................. 2796.52 |=========================== 3sas_zfs_z2_txg_inflight_32 .............. 2797.91 |=========================== 3sas_zfs_z2_txg_128_volthrds_64 .......... 2750.93 |=========================== 1sas_exp_zfs_z2_txg_128_volthrds_64 ...... 2751.86 |=========================== 1sas_exp_zfs_z2_prefetch_rw_2_other_opts . 2647.08 |========================== sync_off_1sas_zfs_z2_prefetch_rw_2_other . 2443.74 |======================== sync_dva_1sas_z2_128_inflight_64_thrds ... 2451.62 |======================== 1sas_all_opts ............................ 2515.09 |======================== 1sas_all_opts_run2 ....................... 2391.45 |======================= 1sas_all_deadline ........................ 2694.19 |========================== 1sas_all_opts_deadline ................... 2449.32 |======================== IOzone 3.465 Record Size: 4Kb - File Size: 2GB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts ............................ 1163.72 |=========================== 3sas_zfs_opts_all_tests .................. 1179.34 |=========================== 3sas_zfs_z2_txg_inflight_32 .............. 1168.99 |=========================== 3sas_zfs_z2_txg_128_volthrds_64 .......... 1159.48 |=========================== 1sas_exp_zfs_z2_txg_128_volthrds_64 ...... 1148.54 |========================== 1sas_exp_zfs_z2_prefetch_rw_2_other_opts . 1161.19 |=========================== sync_off_1sas_zfs_z2_prefetch_rw_2_other . 1146.45 |========================== sync_dva_1sas_z2_128_inflight_64_thrds ... 1140.44 |========================== 1sas_all_opts ............................ 1152.08 |========================== 1sas_all_opts_run2 ....................... 1152.68 |========================== 1sas_all_deadline ........................ 1165.44 |=========================== 1sas_all_opts_deadline ................... 1130.65 |========================== IOzone 3.465 Record Size: 1MB - File Size: 2GB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts ............................ 1288.97 |================ 3sas_zfs_opts_all_tests .................. 1137.12 |============== 3sas_zfs_z2_txg_inflight_32 .............. 1337.84 |================ 3sas_zfs_z2_txg_128_volthrds_64 .......... 1350.50 |================ 1sas_exp_zfs_z2_txg_128_volthrds_64 ...... 1381.80 |================= 1sas_exp_zfs_z2_prefetch_rw_2_other_opts . 1291.30 |================ sync_off_1sas_zfs_z2_prefetch_rw_2_other . 2072.87 |========================= sync_dva_1sas_z2_128_inflight_64_thrds ... 2206.61 |=========================== 1sas_all_opts ............................ 2126.22 |========================== 1sas_all_opts_run2 ....................... 2218.22 |=========================== 1sas_all_deadline ........................ 1411.25 |================= 1sas_all_opts_deadline ................... 2193.61 |=========================== IOzone 3.465 Record Size: 4Kb - File Size: 2GB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts ............................ 411.13 |========================= 3sas_zfs_opts_all_tests .................. 406.69 |======================== 3sas_zfs_z2_txg_inflight_32 .............. 403.05 |======================== 3sas_zfs_z2_txg_128_volthrds_64 .......... 408.30 |========================= 1sas_exp_zfs_z2_txg_128_volthrds_64 ...... 409.14 |========================= 1sas_exp_zfs_z2_prefetch_rw_2_other_opts . 409.75 |========================= sync_off_1sas_zfs_z2_prefetch_rw_2_other . 456.63 |=========================== sync_dva_1sas_z2_128_inflight_64_thrds ... 461.92 |============================ 1sas_all_opts ............................ 455.01 |=========================== 1sas_all_opts_run2 ....................... 465.76 |============================ 1sas_all_deadline ........................ 413.36 |========================= 1sas_all_opts_deadline ................... 456.38 |=========================== IOzone 3.465 Record Size: 64Kb - File Size: 2GB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts ............................ 2693.47 |========================= 3sas_zfs_opts_all_tests .................. 2568.81 |======================== 3sas_zfs_z2_txg_inflight_32 .............. 2906.12 |=========================== 3sas_zfs_z2_txg_128_volthrds_64 .......... 2746.75 |========================== 1sas_exp_zfs_z2_txg_128_volthrds_64 ...... 2738.77 |========================= 1sas_exp_zfs_z2_prefetch_rw_2_other_opts . 2870.51 |=========================== sync_off_1sas_zfs_z2_prefetch_rw_2_other . 2548.72 |======================== sync_dva_1sas_z2_128_inflight_64_thrds ... 2559.49 |======================== 1sas_all_opts ............................ 2783.86 |========================== 1sas_all_opts_run2 ....................... 2665.86 |========================= 1sas_all_opts_deadline ................... 2488.72 |======================= IOzone 3.465 Record Size: 64Kb - File Size: 2GB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts ............................ 1036.24 |==================== 3sas_zfs_opts_all_tests .................. 932.72 |================== 3sas_zfs_z2_txg_inflight_32 .............. 1075.62 |===================== 3sas_zfs_z2_txg_128_volthrds_64 .......... 1040.77 |==================== 1sas_exp_zfs_z2_txg_128_volthrds_64 ...... 1086.85 |===================== 1sas_exp_zfs_z2_prefetch_rw_2_other_opts . 1093.30 |====================== sync_off_1sas_zfs_z2_prefetch_rw_2_other . 1325.67 |========================== sync_dva_1sas_z2_128_inflight_64_thrds ... 1349.36 |=========================== 1sas_all_opts ............................ 1350.65 |=========================== 1sas_all_opts_run2 ....................... 1325.56 |========================== 1sas_all_opts_deadline ................... 1371.30 |=========================== IOzone 3.465 Record Size: 1MB - File Size: 4GB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 2738.57 |============================================ 1sas_all_opts_run2 ...... 2598.80 |========================================== IOzone 3.465 Record Size: 1MB - File Size: 8GB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 2721.32 |============================================ 1sas_all_opts_run2 ...... 2722.78 |============================================ IOzone 3.465 Record Size: 4Kb - File Size: 4GB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 1178.72 |============================================ 1sas_all_opts_run2 ...... 1141.95 |=========================================== IOzone 3.465 Record Size: 4Kb - File Size: 8GB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 1168.91 |============================================ 1sas_all_opts_run2 ...... 1172.22 |============================================ IOzone 3.465 Record Size: 1MB - File Size: 4GB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 1588.41 |================================== 1sas_all_opts_run2 ...... 2076.47 |============================================ IOzone 3.465 Record Size: 1MB - File Size: 8GB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 1739.93 |===================================== 1sas_all_opts_run2 ...... 2052.31 |============================================ IOzone 3.465 Record Size: 4Kb - File Size: 4GB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 417.36 |======================================== 1sas_all_opts_run2 ...... 465.28 |============================================= IOzone 3.465 Record Size: 4Kb - File Size: 8GB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 417.61 |========================================= 1sas_all_opts_run2 ...... 463.85 |============================================= IOzone 3.465 Record Size: 64Kb - File Size: 4GB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 2751.03 |============================================ 1sas_all_opts_run2 ...... 2508.90 |======================================== IOzone 3.465 Record Size: 64Kb - File Size: 8GB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 2845.78 |============================================ 1sas_all_opts_run2 ...... 2677.06 |========================================= IOzone 3.465 Record Size: 1MB - File Size: 512MB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 2613.48 |============================================ 1sas_all_opts_run2 ...... 2487.79 |========================================== IOzone 3.465 Record Size: 4Kb - File Size: 512MB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 1189.00 |============================================ 1sas_all_opts_run2 ...... 1141.72 |========================================== IOzone 3.465 Record Size: 64Kb - File Size: 4GB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 1172.90 |======================================= 1sas_all_opts_run2 ...... 1330.81 |============================================ IOzone 3.465 Record Size: 64Kb - File Size: 8GB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 1227.27 |======================================== 1sas_all_opts_run2 ...... 1364.61 |============================================ IOzone 3.465 Record Size: 1MB - File Size: 512MB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 756.05 |================= 1sas_all_opts_run2 ...... 2008.43 |============================================ IOzone 3.465 Record Size: 4Kb - File Size: 512MB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 372.47 |===================================== 1sas_all_opts_run2 ...... 459.08 |============================================= IOzone 3.465 Record Size: 64Kb - File Size: 512MB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 2877.05 |============================================ 1sas_all_opts_run2 ...... 2528.66 |======================================= IOzone 3.465 Record Size: 64Kb - File Size: 512MB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 717.72 |====================== 1sas_all_opts_run2 ...... 1412.99 |============================================