TrueOS ZoF vs. FreeBSD ZFS vs. ZFS On Linux, Etc Intel Xeon E3-1280 v5 testing of ZFS on BSDs and Linux among other OS file-system tests by Michael Larabel. TrueOS ZoF: Processor: Intel Xeon E3-1280 v5 @ 3.70GHz (8 Cores), Motherboard: MSI MS-7998 1.0, Chipset: Intel Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Registers, Memory: 32768MB, Disk: Generic NVMe Device, Audio: Intel Sunrise Point HDA, Network: Intel PRO/1000 Connection OS: FreeBSD, Kernel: 13.0-CURRENT (x86_64), Compiler: Clang 7.0.1 (SVN 349250), File-System: zfs FreeBSD 12 ZFS: Processor: Intel Xeon E3-1280 v5 @ 3.70GHz (8 Cores), Motherboard: MSI MS-7998 1.0, Chipset: Intel Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Registers, Memory: 32768MB, Disk: Generic NVMe Device, Audio: Intel Sunrise Point HDA, Network: Intel PRO/1000 Connection OS: FreeBSD, Kernel: 12.0-RELEASE (x86_64), Compiler: Clang 6.0.1 (SVN 335540), File-System: zfs, Screen Resolution: 1024x768 DragonFlyBSD 5.4 HAMMER2: Processor: Intel Xeon E3-1280 v5 @ 3.70GHz (4 Cores / 8 Threads), Motherboard: MSI MS-7998 1.0, Chipset: Intel Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Registers, Memory: 32768MB, Disk: NVME-PCIe, Network: Intel PRO/1000 Connection PCH_SPT_I219_V2 7.6.2 OS: DragonFly, Kernel: 5.4-RELEASE (x86_64), Compiler: GCC 8.1, File-System: hammer2 Ubuntu 19.04 EXT4: Processor: Intel Xeon E3-1280 v5 @ 4.00GHz (4 Cores / 8 Threads), Motherboard: MSI Z170A SLI PLUS (MS-7998) v1.0 (2.90 BIOS), Chipset: Intel Xeon E3-1200 v5/E3-1500, Memory: 32768MB, Disk: 256GB TOSHIBA-RD400, Graphics: ASUS AMD Radeon HD 7850 / R7 265 R9 270 1024SP, Audio: Realtek ALC1150, Monitor: VE228, Network: Intel I219-V OS: Ubuntu 19.04, Kernel: 4.18.0-11-generic (x86_64), Desktop: GNOME Shell 3.30.2, Display Server: X Server 1.20.3, Display Driver: modesetting 1.20.3, Compiler: GCC 8.2.0, File-System: ext4, Screen Resolution: 1920x1080 Ubuntu 19.04 ZOL: Processor: Intel Xeon E3-1280 v5 @ 4.00GHz (4 Cores / 8 Threads), Motherboard: MSI Z170A SLI PLUS (MS-7998) v1.0 (2.90 BIOS), Chipset: Intel Xeon E3-1200 v5/E3-1500, Memory: 32768MB, Disk: 256GB TOSHIBA-RD400, Graphics: ASUS AMD Radeon HD 7850 / R7 265 R9 270 1024SP, Audio: Realtek ALC1150, Monitor: VE228, Network: Intel I219-V OS: Ubuntu 19.04, Kernel: 4.18.0-11-generic (x86_64), Desktop: GNOME Shell 3.30.2, Display Server: X Server 1.20.3, Display Driver: modesetting 1.20.3, Compiler: GCC 8.2.0, File-System: zfs, Screen Resolution: 1920x1080 Ubuntu 19.04 F2FS: Processor: Intel Xeon E3-1280 v5 @ 4.00GHz (4 Cores / 8 Threads), Motherboard: MSI Z170A SLI PLUS (MS-7998) v1.0 (2.90 BIOS), Chipset: Intel Xeon E3-1200 v5/E3-1500, Memory: 32768MB, Disk: 256GB TOSHIBA-RD400, Graphics: ASUS AMD Radeon HD 7850 / R7 265 R9 270 1024SP, Audio: Realtek ALC1150, Monitor: VE228, Network: Intel I219-V OS: Ubuntu 19.04, Kernel: 4.18.0-11-generic (x86_64), Desktop: GNOME Shell 3.30.2, Display Server: X Server 1.20.3, Display Driver: modesetting 1.20.3, Compiler: GCC 8.2.0, File-System: f2fs, Screen Resolution: 1920x1080 Clear Linux 27460 EXT4: Processor: Intel Xeon E3-1280 v5 @ 4.00GHz (4 Cores / 8 Threads), Motherboard: MSI Z170A SLI PLUS (MS-7998) v1.0 (2.90 BIOS), Chipset: Intel Xeon E3-1200 v5/E3-1500, Memory: 32768MB, Disk: 256GB TOSHIBA-RD400, Graphics: ASUS AMD Radeon HD 7850 / R7 265 R9 270 1024SP, Audio: Realtek ALC1150, Monitor: VE228, Network: Intel I219-V OS: Clear Linux OS 27460, Kernel: 4.20.4-686.native (x86_64), Desktop: GNOME Shell 3.30.2, Display Server: X Server 1.20.3, Display Driver: modesetting 1.20.3, Compiler: GCC 8.2.1 20180502 + Clang 7.0.1 + LLVM 7.0.1, File-System: ext4, Screen Resolution: 1920x1080 SQLite 3.22 Timed SQLite Insertions Seconds < Lower Is Better TrueOS ZoF ............... 59.10 |============================ FreeBSD 12 ZFS ........... 50.21 |======================== DragonFlyBSD 5.4 HAMMER2 . 4.06 |== Ubuntu 19.04 EXT4 ........ 64.22 |============================== Ubuntu 19.04 ZOL ......... 68.88 |================================ Ubuntu 19.04 F2FS ........ 95.75 |============================================= Clear Linux 27460 EXT4 ... 66.17 |=============================== Flexible IO Tester 3.1 Type: Random Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better TrueOS ZoF ............. 1116.00 |================================ FreeBSD 12 ZFS ......... 1571.00 |============================================= Ubuntu 19.04 EXT4 ...... 37.73 |= Ubuntu 19.04 EXT4 ...... 37.83 |= Ubuntu 19.04 F2FS ...... 37.03 |= Ubuntu 19.04 F2FS ...... 37.50 |= Clear Linux 27460 EXT4 . 25.37 |= Clear Linux 27460 EXT4 . 25.43 |= Flexible IO Tester 3.1 Type: Random Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better TrueOS ZoF ..... 285667 |====================================== FreeBSD 12 ZFS . 402333 |====================================================== Flexible IO Tester 3.1 Type: Random Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better TrueOS ZoF ............. 2098 |================ FreeBSD 12 ZFS ......... 6279 |================================================ Ubuntu 19.04 EXT4 ...... 1001 |======== Ubuntu 19.04 F2FS ...... 998 |======== Clear Linux 27460 EXT4 . 751 |====== Flexible IO Tester 3.1 Type: Random Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better TrueOS ZoF ............. 83.78 |==== FreeBSD 12 ZFS ......... 1018.00 |============================================= Ubuntu 19.04 EXT4 ...... 234.00 |========== Ubuntu 19.04 F2FS ...... 222.00 |========== Clear Linux 27460 EXT4 . 164.00 |======= Flexible IO Tester 3.1 Type: Random Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better TrueOS ZoF ............. 21442 |==== FreeBSD 12 ZFS ......... 260667 |============================================== Ubuntu 19.04 EXT4 ...... 59867 |=========== Ubuntu 19.04 F2FS ...... 56500 |========== Clear Linux 27460 EXT4 . 42000 |======= Flexible IO Tester 3.1 Type: Sequential Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better TrueOS ZoF ............. 3751 |================================== FreeBSD 12 ZFS ......... 5237 |================================================ Ubuntu 19.04 EXT4 ...... 647 |====== Ubuntu 19.04 F2FS ...... 659 |====== Clear Linux 27460 EXT4 . 657 |====== Flexible IO Tester 3.1 Type: Sequential Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better TrueOS ZoF ............. 881.00 |=========================== FreeBSD 12 ZFS ......... 1456.00 |============================================= Ubuntu 19.04 EXT4 ...... 315.00 |========== Ubuntu 19.04 F2FS ...... 307.00 |========= Clear Linux 27460 EXT4 . 41.60 |= Flexible IO Tester 3.1 Type: Sequential Read - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better TrueOS ZoF ............. 225667 |============================ FreeBSD 12 ZFS ......... 373000 |============================================== Ubuntu 19.04 EXT4 ...... 80533 |========== Ubuntu 19.04 F2FS ...... 78600 |========== Clear Linux 27460 EXT4 . 10667 |= Flexible IO Tester 3.1 Type: Sequential Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory MB/s > Higher Is Better TrueOS ZoF ............. 2263 |================= FreeBSD 12 ZFS ......... 6345 |================================================ Ubuntu 19.04 EXT4 ...... 1088 |======== Ubuntu 19.04 F2FS ...... 1106 |======== Clear Linux 27460 EXT4 . 781 |====== Flexible IO Tester 3.1 Type: Sequential Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory MB/s > Higher Is Better TrueOS ZoF ............. 106 |===== FreeBSD 12 ZFS ......... 1049 |================================================ Ubuntu 19.04 EXT4 ...... 250 |=========== Ubuntu 19.04 F2FS ...... 258 |============ Clear Linux 27460 EXT4 . 165 |======== Flexible IO Tester 3.1 Type: Sequential Write - IO Engine: POSIX AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory IOPS > Higher Is Better TrueOS ZoF ............. 27000 |===== FreeBSD 12 ZFS ......... 268667 |============================================== Ubuntu 19.04 EXT4 ...... 64100 |=========== Ubuntu 19.04 F2FS ...... 66100 |=========== Clear Linux 27460 EXT4 . 42267 |======= IOzone 3.465 Record Size: 1MB - File Size: 8GB - Disk Test: Write Performance MB/s > Higher Is Better TrueOS ZoF ............... 1679 |============= FreeBSD 12 ZFS ........... 5933 |============================================== DragonFlyBSD 5.4 HAMMER2 . 1447 |=========== Ubuntu 19.04 EXT4 ........ 957 |======= Ubuntu 19.04 ZOL ......... 718 |====== Ubuntu 19.04 F2FS ........ 1039 |======== Clear Linux 27460 EXT4 ... 862 |======= Compile Bench 0.6 Test: Compile MB/s > Higher Is Better TrueOS ZoF ............... 739 |============ FreeBSD 12 ZFS ........... 2601 |========================================== DragonFlyBSD 5.4 HAMMER2 . 661 |=========== Ubuntu 19.04 EXT4 ........ 2100 |================================== Ubuntu 19.04 ZOL ......... 1136 |================== Ubuntu 19.04 F2FS ........ 2828 |============================================== Clear Linux 27460 EXT4 ... 1958 |================================ Compile Bench 0.6 Test: Initial Create MB/s > Higher Is Better TrueOS ZoF ............... 82.02 |====== FreeBSD 12 ZFS ........... 271.19 |=================== DragonFlyBSD 5.4 HAMMER2 . 102.14 |======= Ubuntu 19.04 EXT4 ........ 556.60 |======================================= Ubuntu 19.04 ZOL ......... 234.93 |================= Ubuntu 19.04 F2FS ........ 621.13 |============================================ Clear Linux 27460 EXT4 ... 505.56 |==================================== Compile Bench 0.6 Test: Read Compiled Tree MB/s > Higher Is Better TrueOS ZoF ............... 315 |===== FreeBSD 12 ZFS ........... 1212 |================== DragonFlyBSD 5.4 HAMMER2 . 961 |============== Ubuntu 19.04 EXT4 ........ 2928 |=========================================== Ubuntu 19.04 ZOL ......... 1391 |==================== Ubuntu 19.04 F2FS ........ 3152 |============================================== Clear Linux 27460 EXT4 ... 3022 |============================================ PostgreSQL pgbench 10.3 Scaling: Buffer Test - Test: Normal Load - Mode: Read Only TPS > Higher Is Better TrueOS ZoF ............... 36555 |============== FreeBSD 12 ZFS ........... 83814 |================================ DragonFlyBSD 5.4 HAMMER2 . 95244 |==================================== Ubuntu 19.04 EXT4 ........ 111396 |========================================== Ubuntu 19.04 ZOL ......... 40070 |=============== Ubuntu 19.04 F2FS ........ 110731 |========================================== Clear Linux 27460 EXT4 ... 115850 |============================================ PostgreSQL pgbench 10.3 Scaling: Buffer Test - Test: Normal Load - Mode: Read Write TPS > Higher Is Better TrueOS ZoF ............... 2333 |============== FreeBSD 12 ZFS ........... 2451 |=============== DragonFlyBSD 5.4 HAMMER2 . 7473 |============================================== Ubuntu 19.04 EXT4 ........ 3477 |===================== Ubuntu 19.04 ZOL ......... 1235 |======== Ubuntu 19.04 F2FS ........ 3342 |===================== Clear Linux 27460 EXT4 ... 3083 |=================== BlogBench 1.1 Test: Read Final Score > Higher Is Better TrueOS ZoF ............... 547697 |================ FreeBSD 12 ZFS ........... 1363053 |======================================== DragonFlyBSD 5.4 HAMMER2 . 617430 |================== Ubuntu 19.04 EXT4 ........ 1337223 |======================================== Ubuntu 19.04 ZOL ......... 516671 |=============== Ubuntu 19.04 F2FS ........ 1447329 |=========================================== Clear Linux 27460 EXT4 ... 1283725 |====================================== BlogBench 1.1 Test: Write Final Score > Higher Is Better TrueOS ZoF ............... 123 | FreeBSD 12 ZFS ........... 3633 |======= DragonFlyBSD 5.4 HAMMER2 . 5506 |========== Ubuntu 19.04 EXT4 ........ 23604 |============================================= Ubuntu 19.04 ZOL ......... 5219 |========== Ubuntu 19.04 F2FS ........ 13411 |========================== Clear Linux 27460 EXT4 ... 15249 |=============================