dbox 2 x Intel Xeon L5640 testing with a Supermicro X8DTH and Matrox s MGA G200eW WPCM450 on Ubuntu 16.04 via the Phoronix Test Suite. 3sas_zfs_opts: Processor: 2 x Intel Xeon L5640 @ 2.27GHz (24 Cores), Motherboard: Supermicro X8DTH, Chipset: Intel 5520 I/O + ICH10R, Memory: 80896MB, Disk: 500GB Western Digital WD5000ABYS-0 + 500GB Seagate ST9500420ASG + 2 x 1000GB Seagate ST1000LM014-1EJ1 + 4 x 4001GB TOSHIBA HDWQ140 + 1000GB Western Digital WD10EZEX-00B + 2 x 4001GB HGST HDN726040AL + 64GB Type-C + 8GB Cruzer Blade + 3 x 1000GB Seagate ST31000528AS + Samsung SSD 950 PRO 512GB, Graphics: Matrox s MGA G200eW WPCM450, Network: Intel 82576 Gigabit Connection + Mellanox MT27500 Family OS: Ubuntu 16.04, Kernel: 4.10.0-35-lowlatency (x86_64), Compiler: GCC 5.4.0 20160609, File-System: zfs, Screen Resolution: 1024x768 3sas_zfs_opts_all_tests: Processor: 2 x Intel Xeon L5640 @ 2.27GHz (24 Cores), Motherboard: Supermicro X8DTH, Chipset: Intel 5520 I/O + ICH10R, Memory: 80896MB, Disk: 500GB Western Digital WD5000ABYS-0 + 500GB Seagate ST9500420ASG + 2 x 1000GB Seagate ST1000LM014-1EJ1 + 4 x 4001GB TOSHIBA HDWQ140 + 1000GB Western Digital WD10EZEX-00B + 2 x 4001GB HGST HDN726040AL + 64GB Type-C + 8GB Cruzer Blade + 3 x 1000GB Seagate ST31000528AS + Samsung SSD 950 PRO 512GB, Graphics: Matrox s MGA G200eW WPCM450, Network: Intel 82576 Gigabit Connection + Mellanox MT27500 Family OS: Ubuntu 16.04, Kernel: 4.10.0-35-lowlatency (x86_64), Compiler: GCC 5.4.0 20160609, File-System: zfs, Screen Resolution: 1024x768 3sas_zfs_z2_txg_inflight_32: Processor: 2 x Intel Xeon L5640 @ 2.27GHz (24 Cores), Motherboard: Supermicro X8DTH, Chipset: Intel 5520 I/O + ICH10R, Memory: 80896MB, Disk: 500GB Western Digital WD5000ABYS-0 + 500GB Seagate ST9500420ASG + 2 x 1000GB Seagate ST1000LM014-1EJ1 + 4 x 4001GB TOSHIBA HDWQ140 + 1000GB Western Digital WD10EZEX-00B + 2 x 4001GB HGST HDN726040AL + 64GB Type-C + 8GB Cruzer Blade + 3 x 1000GB Seagate ST31000528AS + Samsung SSD 950 PRO 512GB, Graphics: Matrox s MGA G200eW WPCM450, Network: Intel 82576 Gigabit Connection + Mellanox MT27500 Family OS: Ubuntu 16.04, Kernel: 4.10.0-35-lowlatency (x86_64), Compiler: GCC 5.4.0 20160609, File-System: zfs, Screen Resolution: 1024x768 3sas_zfs_z2_txg_128_volthrds_64: Processor: 2 x Intel Xeon L5640 @ 2.27GHz (24 Cores), Motherboard: Supermicro X8DTH, Chipset: Intel 5520 I/O + ICH10R, Memory: 80896MB, Disk: 500GB Western Digital WD5000ABYS-0 + 500GB Seagate ST9500420ASG + 2 x 1000GB Seagate ST1000LM014-1EJ1 + 4 x 4001GB TOSHIBA HDWQ140 + 1000GB Western Digital WD10EZEX-00B + 2 x 4001GB HGST HDN726040AL + 64GB Type-C + 8GB Cruzer Blade + 3 x 1000GB Seagate ST31000528AS + Samsung SSD 950 PRO 512GB, Graphics: Matrox s MGA G200eW WPCM450, Network: Intel 82576 Gigabit Connection + Mellanox MT27500 Family OS: Ubuntu 16.04, Kernel: 4.10.0-35-lowlatency (x86_64), Compiler: GCC 5.4.0 20160609, File-System: zfs, Screen Resolution: 1024x768 1sas_exp_zfs_z2_txg_128_volthrds_64: Processor: 2 x Intel Xeon L5640 @ 2.27GHz (24 Cores), Motherboard: Supermicro X8DTH, Chipset: Intel 5520 I/O + ICH10R, Memory: 80896MB, Disk: 8GB Cruzer Blade + 64GB Type-C + 3 x 1000GB Seagate ST31000528AS + 2 x 1000GB Seagate ST1000LM014-1EJ1 + 4 x 4001GB TOSHIBA HDWQ140 + 500GB Western Digital WD5000ABYS-0 + 500GB Seagate ST9500420ASG + 1000GB Western Digital WD10EZEX-00B + 2 x 4001GB HGST HDN726040AL + Samsung SSD 950 PRO 512GB, Graphics: Matrox s MGA G200eW WPCM450, Network: Intel 82576 Gigabit Connection + Mellanox MT27500 Family OS: Ubuntu 16.04, Kernel: 4.10.0-35-lowlatency (x86_64), Compiler: GCC 5.4.0 20160609, File-System: zfs, Screen Resolution: 1024x768 1sas_exp_zfs_z2_prefetch_rw_2_other_opts: Processor: 2 x Intel Xeon L5640 @ 2.27GHz (24 Cores), Motherboard: Supermicro X8DTH, Chipset: Intel 5520 I/O + ICH10R, Memory: 80896MB, Disk: 8GB Cruzer Blade + 64GB Type-C + 3 x 1000GB Seagate ST31000528AS + 2 x 1000GB Seagate ST1000LM014-1EJ1 + 4 x 4001GB TOSHIBA HDWQ140 + 500GB Western Digital WD5000ABYS-0 + 500GB Seagate ST9500420ASG + 1000GB Western Digital WD10EZEX-00B + 2 x 4001GB HGST HDN726040AL + Samsung SSD 950 PRO 512GB, Graphics: Matrox s MGA G200eW WPCM450, Network: Intel 82576 Gigabit Connection + Mellanox MT27500 Family OS: Ubuntu 16.04, Kernel: 4.10.0-35-lowlatency (x86_64), Compiler: GCC 5.4.0 20160609, File-System: zfs, Screen Resolution: 1024x768 sync_off_1sas_zfs_z2_prefetch_rw_2_other: Processor: 2 x Intel Xeon L5640 @ 2.27GHz (24 Cores), Motherboard: Supermicro X8DTH, Chipset: Intel 5520 I/O + ICH10R, Memory: 80896MB, Disk: 8GB Cruzer Blade + 64GB Type-C + 3 x 1000GB Seagate ST31000528AS + 2 x 1000GB Seagate ST1000LM014-1EJ1 + 4 x 4001GB TOSHIBA HDWQ140 + 500GB Western Digital WD5000ABYS-0 + 500GB Seagate ST9500420ASG + 1000GB Western Digital WD10EZEX-00B + 2 x 4001GB HGST HDN726040AL + Samsung SSD 950 PRO 512GB, Graphics: Matrox s MGA G200eW WPCM450, Network: Intel 82576 Gigabit Connection + Mellanox MT27500 Family OS: Ubuntu 16.04, Kernel: 4.10.0-35-lowlatency (x86_64), Compiler: GCC 5.4.0 20160609, File-System: zfs, Screen Resolution: 1024x768 sync_dva_1sas_z2_128_inflight_64_thrds: Processor: 2 x Intel Xeon L5640 @ 2.27GHz (24 Cores), Motherboard: Supermicro X8DTH, Chipset: Intel 5520 I/O + ICH10R, Memory: 80896MB, Disk: 8GB Cruzer Blade + 64GB Type-C + 3 x 1000GB Seagate ST31000528AS + 2 x 1000GB Seagate ST1000LM014-1EJ1 + 4 x 4001GB TOSHIBA HDWQ140 + 500GB Western Digital WD5000ABYS-0 + 500GB Seagate ST9500420ASG + 1000GB Western Digital WD10EZEX-00B + 2 x 4001GB HGST HDN726040AL + Samsung SSD 950 PRO 512GB, Graphics: Matrox s MGA G200eW WPCM450, Network: Intel 82576 Gigabit Connection + Mellanox MT27500 Family OS: Ubuntu 16.04, Kernel: 4.10.0-35-lowlatency (x86_64), Compiler: GCC 5.4.0 20160609, File-System: zfs, Screen Resolution: 1024x768 IOzone 3.465 Record Size: 1MB - File Size: 2GB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts ............................ 2801.53 |=========================== 3sas_zfs_opts_all_tests .................. 2796.52 |=========================== 3sas_zfs_z2_txg_inflight_32 .............. 2797.91 |=========================== 3sas_zfs_z2_txg_128_volthrds_64 .......... 2750.93 |=========================== 1sas_exp_zfs_z2_txg_128_volthrds_64 ...... 2751.86 |=========================== 1sas_exp_zfs_z2_prefetch_rw_2_other_opts . 2647.08 |========================== sync_off_1sas_zfs_z2_prefetch_rw_2_other . 2443.74 |======================== sync_dva_1sas_z2_128_inflight_64_thrds ... 2451.62 |======================== IOzone 3.465 Record Size: 4Kb - File Size: 2GB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts ............................ 1163.72 |=========================== 3sas_zfs_opts_all_tests .................. 1179.34 |=========================== 3sas_zfs_z2_txg_inflight_32 .............. 1168.99 |=========================== 3sas_zfs_z2_txg_128_volthrds_64 .......... 1159.48 |=========================== 1sas_exp_zfs_z2_txg_128_volthrds_64 ...... 1148.54 |========================== 1sas_exp_zfs_z2_prefetch_rw_2_other_opts . 1161.19 |=========================== sync_off_1sas_zfs_z2_prefetch_rw_2_other . 1146.45 |========================== sync_dva_1sas_z2_128_inflight_64_thrds ... 1140.44 |========================== IOzone 3.465 Record Size: 1MB - File Size: 2GB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts ............................ 1288.97 |================ 3sas_zfs_opts_all_tests .................. 1137.12 |============== 3sas_zfs_z2_txg_inflight_32 .............. 1337.84 |================ 3sas_zfs_z2_txg_128_volthrds_64 .......... 1350.50 |================= 1sas_exp_zfs_z2_txg_128_volthrds_64 ...... 1381.80 |================= 1sas_exp_zfs_z2_prefetch_rw_2_other_opts . 1291.30 |================ sync_off_1sas_zfs_z2_prefetch_rw_2_other . 2072.87 |========================= sync_dva_1sas_z2_128_inflight_64_thrds ... 2206.61 |=========================== IOzone 3.465 Record Size: 4Kb - File Size: 2GB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts ............................ 411.13 |========================= 3sas_zfs_opts_all_tests .................. 406.69 |========================= 3sas_zfs_z2_txg_inflight_32 .............. 403.05 |======================== 3sas_zfs_z2_txg_128_volthrds_64 .......... 408.30 |========================= 1sas_exp_zfs_z2_txg_128_volthrds_64 ...... 409.14 |========================= 1sas_exp_zfs_z2_prefetch_rw_2_other_opts . 409.75 |========================= sync_off_1sas_zfs_z2_prefetch_rw_2_other . 456.63 |============================ sync_dva_1sas_z2_128_inflight_64_thrds ... 461.92 |============================ IOzone 3.465 Record Size: 64Kb - File Size: 2GB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts ............................ 2693.47 |========================= 3sas_zfs_opts_all_tests .................. 2568.81 |======================== 3sas_zfs_z2_txg_inflight_32 .............. 2906.12 |=========================== 3sas_zfs_z2_txg_128_volthrds_64 .......... 2746.75 |========================== 1sas_exp_zfs_z2_txg_128_volthrds_64 ...... 2738.77 |========================= 1sas_exp_zfs_z2_prefetch_rw_2_other_opts . 2870.51 |=========================== sync_off_1sas_zfs_z2_prefetch_rw_2_other . 2548.72 |======================== sync_dva_1sas_z2_128_inflight_64_thrds ... 2559.49 |======================== IOzone 3.465 Record Size: 64Kb - File Size: 2GB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts ............................ 1036.24 |===================== 3sas_zfs_opts_all_tests .................. 932.72 |=================== 3sas_zfs_z2_txg_inflight_32 .............. 1075.62 |====================== 3sas_zfs_z2_txg_128_volthrds_64 .......... 1040.77 |===================== 1sas_exp_zfs_z2_txg_128_volthrds_64 ...... 1086.85 |====================== 1sas_exp_zfs_z2_prefetch_rw_2_other_opts . 1093.30 |====================== sync_off_1sas_zfs_z2_prefetch_rw_2_other . 1325.67 |=========================== sync_dva_1sas_z2_128_inflight_64_thrds ... 1349.36 |=========================== IOzone 3.465 Record Size: 1MB - File Size: 4GB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 2738.57 |============================================ IOzone 3.465 Record Size: 1MB - File Size: 8GB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 2721.32 |============================================ IOzone 3.465 Record Size: 4Kb - File Size: 4GB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 1178.72 |============================================ IOzone 3.465 Record Size: 4Kb - File Size: 8GB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 1168.91 |============================================ IOzone 3.465 Record Size: 1MB - File Size: 4GB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 1588.41 |============================================ IOzone 3.465 Record Size: 1MB - File Size: 8GB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 1739.93 |============================================ IOzone 3.465 Record Size: 4Kb - File Size: 4GB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 417.36 |============================================= IOzone 3.465 Record Size: 4Kb - File Size: 8GB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 417.61 |============================================= IOzone 3.465 Record Size: 64Kb - File Size: 4GB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 2751.03 |============================================ IOzone 3.465 Record Size: 64Kb - File Size: 8GB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 2845.78 |============================================ IOzone 3.465 Record Size: 1MB - File Size: 512MB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 2613.48 |============================================ IOzone 3.465 Record Size: 4Kb - File Size: 512MB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 1189.00 |============================================ IOzone 3.465 Record Size: 64Kb - File Size: 4GB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 1172.90 |============================================ IOzone 3.465 Record Size: 64Kb - File Size: 8GB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 1227.27 |============================================ IOzone 3.465 Record Size: 1MB - File Size: 512MB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 756.05 |============================================= IOzone 3.465 Record Size: 4Kb - File Size: 512MB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 372.47 |============================================= IOzone 3.465 Record Size: 64Kb - File Size: 512MB - Disk Test: Read Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 2877.05 |============================================ IOzone 3.465 Record Size: 64Kb - File Size: 512MB - Disk Test: Write Performance MB/s > Higher Is Better 3sas_zfs_opts_all_tests . 717.72 |=============================================