ssd Bionic LXD container in Xenial host, ZFS on 2x(4x6TB HDD RAIDZ) Ceph 4SS ODSs, 1Gb eth: Processor: Intel Core i5-5300U @ 2.90GHz (2 Cores / 4 Threads), Motherboard: Intel NUC5i5MYBE (MYBDWi5v.86A.0026.2015.0820.1501 BIOS), Chipset: Intel Broadwell-U-OPI, Memory: 16384MB, Disk: 256GB Samsung SSD 850, Graphics: Intel HD 5500 (900MHz), Audio: Intel Broadwell-U Audio, Network: Intel Connection OS: Ubuntu 18.04, Kernel: 4.15.0-13-generic (x86_64), Compiler: GCC 7.3.0, File-System: ext4, System Layer: lxc DRBD/RAID1/NVMe: Processor: Intel Xeon E3-1280 v6 @ 4.20GHz (4 Cores / 8 Threads), Motherboard: Supermicro X11SSL-F v1.01 (2.0a BIOS), Memory: 64512MB, Disk: 2 x 800GB INTEL SSDSC2BB80 + 2 x 400GB INTEL SSDPEDME400G4, Graphics: astdrmfb OS: Ubuntu 16.04, Kernel: 4.4.0-93-generic (x86_64), Compiler: GCC 5.4.0 20160609, File-System: ext4, Screen Resolution: 1024x768, System Layer: lxc LXD/ZFS/HD: Processor: Intel Xeon E3-1240 v3 @ 3.80GHz (4 Cores / 8 Threads), Motherboard: Supermicro X10SLL-F v1.02 (2.2 BIOS), Chipset: Intel Xeon E3-1200 v3 DRAM, Memory: 32768MB, Disk: 8 x 6001GB Western Digital WD60EFRX-68M + 2 x 120GB INTEL SSDSC2BB12, Graphics: ASPEED ASPEED Family, Network: Intel Connection I217-LM OS: Ubuntu 18.04, Kernel: 4.4.0-116-generic (x86_64), Compiler: GCC 7.3.0, File-System: zfs, Screen Resolution: 1024x768, System Layer: lxc SSD: Processor: Intel Core i5-5300U @ 2.90GHz (2 Cores / 4 Threads), Motherboard: Intel NUC5i5MYBE (MYBDWi5v.86A.0026.2015.0820.1501 BIOS), Chipset: Intel Broadwell-U-OPI, Memory: 16384MB, Disk: 512GB Samsung SSD 850, Graphics: Intel HD 5500 (900MHz), Audio: Intel Broadwell-U Audio, Monitor: HP w20, Network: Intel Connection OS: Ubuntu 18.04, Kernel: 4.15.0-13-generic (x86_64), Compiler: GCC 7.3.0, File-System: ext4, Screen Resolution: 1680x1050 NVMe: Processor: Intel Xeon E3-1280 v6 @ 4.20GHz (4 Cores / 8 Threads), Motherboard: Supermicro X11SSL-F v1.01 (2.0a BIOS), Memory: 64512MB, Disk: 2 x 800GB INTEL SSDSC2BB80 + 2 x 400GB INTEL SSDPEDME400G4, Graphics: astdrmfb OS: Ubuntu 18.04, Kernel: 4.4.0-93-generic (x86_64), Compiler: GCC 7.3.0, File-System: ext2, Screen Resolution: 1024x768, System Layer: lxc Apache Benchmark 2.4.29 Static Web Page Serving Requests Per Second > Higher Is Better Ceph 4SS ODSs, 1Gb eth . 14605.66 |===================== DRBD/RAID1/NVMe ........ 30202.25 |============================================ LXD/ZFS/HD ............. 17967.17 |========================== SSD .................... 11945.86 |================= NVMe ................... 30230.49 |============================================ Gzip Compression Linux Source Tree Archiving To .tar.gz Seconds < Lower Is Better Ceph 4SS ODSs, 1Gb eth . 52.77 |=============================================== DRBD/RAID1/NVMe ........ 34.08 |============================== LXD/ZFS/HD ............. 38.56 |================================== SSD .................... 50.16 |============================================= NVMe ................... 36.06 |================================ PostMark 1.51 Disk Transaction Performance TPS > Higher Is Better Ceph 4SS ODSs, 1Gb eth . 4261 |============================ DRBD/RAID1/NVMe ........ 7282 |================================================ LXD/ZFS/HD ............. 3157 |===================== SSD .................... 4807 |================================ NVMe ................... 5708 |====================================== Unpacking The Linux Kernel linux-4.15.tar.xz Seconds < Lower Is Better Ceph 4SS ODSs, 1Gb eth . 9.19 |======================================== DRBD/RAID1/NVMe ........ 5.64 |======================== LXD/ZFS/HD ............. 7.73 |================================= SSD .................... 8.26 |==================================== NVMe ................... 10.85 |=============================================== Compile Bench 0.6 Test: Read Compiled Tree MB/s > Higher Is Better Ceph 4SS ODSs, 1Gb eth . 1060.11 |================== DRBD/RAID1/NVMe ........ 2685.94 |============================================= LXD/ZFS/HD ............. 708.39 |============ SSD .................... 2042.16 |================================== NVMe ................... 2304.86 |======================================= Compile Bench 0.6 Test: Initial Create MB/s > Higher Is Better Ceph 4SS ODSs, 1Gb eth . 187.37 |================= DRBD/RAID1/NVMe ........ 520.65 |============================================== LXD/ZFS/HD ............. 192.40 |================= SSD .................... 350.97 |=============================== NVMe ................... 206.23 |================== Compile Bench 0.6 Test: Compile MB/s > Higher Is Better Ceph 4SS ODSs, 1Gb eth . 436.69 |======== DRBD/RAID1/NVMe ........ 2471.56 |============================================= LXD/ZFS/HD ............. 1167.99 |===================== SSD .................... 862.26 |================ NVMe ................... 1732.33 |================================ Threaded I/O Tester 20170503 64MB Random Write - 32 Threads MB/s > Higher Is Better Ceph 4SS ODSs, 1Gb eth . 62.34 | DRBD/RAID1/NVMe ........ 6206.57 |============================================= LXD/ZFS/HD ............. 42.94 | SSD .................... 93.97 |= NVMe ................... 5618.22 |========================================= Threaded I/O Tester 20170503 64MB Random Read - 32 Threads MB/s > Higher Is Better Ceph 4SS ODSs, 1Gb eth . 47240.68 |======== DRBD/RAID1/NVMe ........ 270736.23 |=========================================== LXD/ZFS/HD ............. 14446.66 |== SSD .................... 73033.05 |============ NVMe ................... 237556.39 |====================================== Dbench 4.0 1 Clients MB/s > Higher Is Better Ceph 4SS ODSs, 1Gb eth . 17.89 |= DRBD/RAID1/NVMe ........ 923.36 |============================================== LXD/ZFS/HD ............. 33.38 |== SSD .................... 42.76 |== NVMe ................... 808.69 |======================================== Dbench 4.0 128 Clients MB/s > Higher Is Better Ceph 4SS ODSs, 1Gb eth . 286.69 |=== DRBD/RAID1/NVMe ........ 2739.20 |=========================== LXD/ZFS/HD ............. 240.45 |== SSD .................... 1010.77 |========== NVMe ................... 4649.74 |============================================= Dbench 4.0 48 Clients MB/s > Higher Is Better Ceph 4SS ODSs, 1Gb eth . 189.28 |== DRBD/RAID1/NVMe ........ 3239.23 |============================== LXD/ZFS/HD ............. 174.95 |== SSD .................... 570.62 |===== NVMe ................... 4928.96 |============================================= Dbench 4.0 12 Clients MB/s > Higher Is Better Ceph 4SS ODSs, 1Gb eth . 86.08 |= DRBD/RAID1/NVMe ........ 3142.73 |=================================== LXD/ZFS/HD ............. 139.65 |== SSD .................... 210.28 |== NVMe ................... 4050.65 |============================================= FS-Mark 3.3 1000 Files, 1MB Size Files/s > Higher Is Better Ceph 4SS ODSs, 1Gb eth . 17.37 |== DRBD/RAID1/NVMe ........ 433.03 |============================================== LXD/ZFS/HD ............. 52.67 |====== SSD .................... 57.90 |====== NVMe ................... 421.70 |============================================= SQLite 3.22 Timed SQLite Insertions Seconds < Lower Is Better Ceph 4SS ODSs, 1Gb eth . 323.25 |============================================== DRBD/RAID1/NVMe ........ 2.54 | LXD/ZFS/HD ............. 269.14 |====================================== SSD .................... 135.45 |=================== NVMe ................... 2.73 | AIO-Stress 0.21 Random Write MB/s > Higher Is Better Ceph 4SS ODSs, 1Gb eth . 2219.44 |============================ DRBD/RAID1/NVMe ........ 3573.95 |============================================= LXD/ZFS/HD ............. 1382.49 |================= SSD .................... 2292.94 |============================= NVMe ................... 1918.11 |========================