10GbE Linux vs. Windows vs. BSD Network Benchmarks
10GbE Linux network performance benchmarks by Michael Larabel for a future article on Phoronix.com.


ConnectX: Windows 2019: 

	Processor: 2 x Intel Xeon Gold 6138 @ 2.00GHz (40 Cores / 80 Threads), Motherboard: TYAN S7106, Memory: 12 x 8192 MB 2666MHz Micron 9ASF1G72PZ-2G6B1, Disk: 233GB Samsung SSD 970 EVO 250GB, Graphics: Microsoft Basic Display, Network: Intel I210 Gigabit Connection + Intel I210 Gigabit Connection #2

	OS: Microsoft Windows Server 2019 Essentials Build 17763, Kernel: 10.0 (x86_64), Display Driver: 10.0.17763.1, File-System: NTFS, Screen Resolution: 1024x768

ConnectX: FreeBSD 12: 

	Processor: Intel Xeon Gold 6138 @ 2.00GHz (80 Cores), Motherboard: TYAN B7106G24EV4HR, Chipset: Intel Sky Lake-E DMI3 Registers, Memory: 97024MB, Disk: Generic NVMe Device, Graphics: ASPEED Family

	OS: FreeBSD, Kernel: 12.0-RELEASE (x86_64), Compiler: Clang 6.0.1 (SVN 335540), File-System: zfs, Screen Resolution: 800x600

ConnectX: Ubuntu 18.04: 

	Processor: 2 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads), Motherboard: TYAN S7106 (V1.02.B20 BIOS), Chipset: Intel Sky Lake-E DMI3 Registers, Memory: 96256MB, Disk: Samsung SSD 970 EVO 250GB, Graphics: ASPEED Family, Monitor: VE228, Network: 2 x Intel I210 + 2 x QLogic cLOM8214 1/10GbE + Mellanox MT26448

	OS: Ubuntu 18.04, Kernel: 4.15.0-43-generic (x86_64), Desktop: GNOME Shell 3.28.3, Display Server: X Server 1.19.6, Display Driver: modesetting 1.19.6, OpenGL: 3.3 Mesa 18.0.5 (LLVM 6.0 256 bits), Compiler: GCC 7.3.0, File-System: ext4, Screen Resolution: 1920x1080

ConnectX: Debian 9.6: 

	Processor: 2 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads), Motherboard: TYAN S7106 (V1.02.B20 BIOS), Chipset: Intel Device 2020, Memory: 12 x 8192 MB DDR4-2666MHz Micron 9ASF1G72PZ-2G6B1, Disk: Samsung SSD 970 EVO 250GB, Graphics: LLVMpipe, Network: 2 x Intel I210 + 2 x QLogic cLOM8214 1/10GbE + Mellanox MT26448

	OS: Debian 9.6, Kernel: 4.9.0-8-amd64 (x86_64), Desktop: GNOME Shell 3.22.3, Display Server: X Server 1.19.2, OpenGL: 3.3 Mesa 13.0.6 Gallium 0.4 (LLVM 3.9 256 bits), Compiler: GCC 6.3.0 20170516, File-System: ext4, Screen Resolution: 1024x768

ConnectX: Ubuntu 18.10: 

	Processor: 2 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads), Motherboard: TYAN S7106 (V1.02.B20 BIOS), Chipset: Intel Sky Lake-E DMI3 Registers, Memory: 96256MB, Disk: Samsung SSD 970 EVO 250GB, Graphics: llvmpipe 93GB, Monitor: VE228, Network: 2 x Intel I210 + 2 x QLogic cLOM8214 1/10GbE + Mellanox MT26448

	OS: Ubuntu 18.10, Kernel: 4.18.0-13-generic (x86_64), Desktop: GNOME Shell 3.30.1, Display Server: X Server 1.20.1, Display Driver: modesetting 1.20.1, OpenGL: 3.3 Mesa 18.2.2 (LLVM 7.0 256 bits), Compiler: GCC 8.2.0, File-System: ext4, Screen Resolution: 1920x1080

ConnectX: Ubuntu 18.10 Tuned: 

	Processor: 2 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads), Motherboard: TYAN S7106 (V1.02.B20 BIOS), Chipset: Intel Sky Lake-E DMI3 Registers, Memory: 96256MB, Disk: Samsung SSD 970 EVO 250GB, Graphics: llvmpipe 93GB, Monitor: VE228, Network: 2 x Intel I210 + 2 x QLogic cLOM8214 1/10GbE + Mellanox MT26448

	OS: Ubuntu 18.10, Kernel: 4.18.0-13-generic (x86_64), Desktop: GNOME Shell 3.30.1, Display Server: X Server 1.20.1, Display Driver: modesetting 1.20.1, OpenGL: 3.3 Mesa 18.2.2 (LLVM 7.0 256 bits), Compiler: GCC 8.2.0, File-System: ext4, Screen Resolution: 1920x1080

ConnectX: Scientific 7: 

	Processor: 2 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads), Motherboard: TYAN S7106 (V1.02.B20 BIOS), Chipset: Intel Sky Lake-E DMI3 Registers, Memory: 12 x 8192 MB DDR4-2666MT/s Micron 9ASF1G72PZ-2G6B1, Disk: Samsung SSD 970 EVO 250GB, Graphics: llvmpipe 93GB, Monitor: VE228, Network: 2 x Intel I210 + 2 x QLogic cLOM8214 1/10GbE + Mellanox MT26448

	OS: Scientific 7.6, Kernel: 3.10.0-957.1.3.el7.x86_64 (x86_64), Desktop: GNOME Shell 3.28.3, Display Server: X Server 1.20.1, Display Driver: modesetting 1.20.1, OpenGL: 2.1 Mesa 18.0.5 (LLVM 6.0 256 bits), Compiler: GCC 4.8.5 20150623, File-System: xfs, Screen Resolution: 1920x1080

ConnectX: Scientific 7 Tuned: 

	Processor: 2 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads), Motherboard: TYAN S7106 (V1.02.B20 BIOS), Chipset: Intel Sky Lake-E DMI3 Registers, Memory: 12 x 8192 MB DDR4-2666MT/s Micron 9ASF1G72PZ-2G6B1, Disk: Samsung SSD 970 EVO 250GB, Graphics: llvmpipe 93GB, Monitor: VE228, Network: 2 x Intel I210 + 2 x QLogic cLOM8214 1/10GbE + Mellanox MT26448

	OS: Scientific 7.6, Kernel: 3.10.0-957.1.3.el7.x86_64 (x86_64), Desktop: GNOME Shell 3.28.3, Display Server: X Server 1.20.1, Display Driver: modesetting 1.20.1, OpenGL: 2.1 Mesa 18.0.5 (LLVM 6.0 256 bits), Compiler: GCC 4.8.5 20150623, File-System: xfs, Screen Resolution: 1920x1080


iPerf 3.1.3
Server Address: 192.168.1.83 - Server Port: 5201 - Duration: 60 Seconds - Test: TCP - Parallel: 1
Mbits/sec > Higher Is Better
ConnectX: Windows 2019 ....... 5672 |==========================
ConnectX: FreeBSD 12 ......... 8640 |========================================
ConnectX: Ubuntu 18.04 ....... 9178 |==========================================
ConnectX: Debian 9.6 ......... 9165 |==========================================
ConnectX: Ubuntu 18.10 ....... 7982 |=====================================
ConnectX: Ubuntu 18.10 Tuned . 7846 |====================================
ConnectX: Scientific 7 ....... 9175 |==========================================
ConnectX: Scientific 7 Tuned . 9173 |==========================================


iPerf 3.1.3
Server Address: 192.168.1.83 - Server Port: 5201 - Duration: 60 Seconds - Test: TCP - Parallel: 5
Mbits/sec > Higher Is Better
ConnectX: Windows 2019 ....... 9252 |==========================================
ConnectX: FreeBSD 12 ......... 8879 |========================================
ConnectX: Ubuntu 18.04 ....... 8862 |========================================
ConnectX: Debian 9.6 ......... 8874 |========================================
ConnectX: Ubuntu 18.10 ....... 6940 |================================
ConnectX: Ubuntu 18.10 Tuned . 7014 |================================
ConnectX: Scientific 7 ....... 8810 |========================================
ConnectX: Scientific 7 Tuned . 8935 |=========================================


iPerf 3.1.3
Server Address: 192.168.1.83 - Server Port: 5201 - Duration: 60 Seconds - Test: TCP - Parallel: 20
Mbits/sec > Higher Is Better
ConnectX: Windows 2019 ....... 9098 |==========================================
ConnectX: FreeBSD 12 ......... 8626 |========================================
ConnectX: Ubuntu 18.04 ....... 8713 |========================================
ConnectX: Debian 9.6 ......... 8750 |========================================
ConnectX: Ubuntu 18.10 ....... 6701 |===============================
ConnectX: Ubuntu 18.10 Tuned . 6709 |===============================
ConnectX: Scientific 7 ....... 8726 |========================================
ConnectX: Scientific 7 Tuned . 8708 |========================================


Nuttcp 8.1.4
Test: TCP Transfer - Default - Server Address: 192.168.1.83
Mbits/sec > Higher Is Better
ConnectX: Windows 2019 ....... 9420 |==========================================
ConnectX: FreeBSD 12 ......... 8909 |========================================
ConnectX: Ubuntu 18.04 ....... 9156 |=========================================
ConnectX: Debian 9.6 ......... 9177 |=========================================
ConnectX: Ubuntu 18.10 ....... 7997 |====================================
ConnectX: Ubuntu 18.10 Tuned . 7909 |===================================
ConnectX: Scientific 7 ....... 9251 |=========================================
ConnectX: Scientific 7 Tuned . 9257 |=========================================


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: TCP - Test: Latency - Threads: 1
Microseconds < Lower Is Better
ConnectX: Windows 2019 ....... 64.10 |=====================================
ConnectX: Ubuntu 18.04 ....... 56.34 |================================
ConnectX: Debian 9.6 ......... 52.67 |==============================
ConnectX: Ubuntu 18.10 ....... 56.82 |=================================
ConnectX: Ubuntu 18.10 Tuned . 47.08 |===========================
ConnectX: Scientific 7 ....... 71.56 |=========================================
ConnectX: Scientific 7 Tuned . 47.12 |===========================


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: TCP - Test: Latency - Threads: 8
Microseconds < Lower Is Better
ConnectX: Windows 2019 ....... 64.94 |=====================================
ConnectX: Ubuntu 18.04 ....... 57.94 |=================================
ConnectX: Debian 9.6 ......... 52.75 |==============================
ConnectX: Ubuntu 18.10 ....... 60.60 |===================================
ConnectX: Ubuntu 18.10 Tuned . 44.62 |==========================
ConnectX: Scientific 7 ....... 71.24 |=========================================
ConnectX: Scientific 7 Tuned . 46.62 |===========================


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: TCP - Test: Latency - Threads: 64
Microseconds < Lower Is Better
ConnectX: Windows 2019 ....... 64.48 |=====================================
ConnectX: Ubuntu 18.04 ....... 52.45 |==============================
ConnectX: Debian 9.6 ......... 52.80 |==============================
ConnectX: Ubuntu 18.10 ....... 57.62 |=================================
ConnectX: Ubuntu 18.10 Tuned . 46.19 |===========================
ConnectX: Scientific 7 ....... 71.45 |=========================================
ConnectX: Scientific 7 Tuned . 45.48 |==========================


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: TCP - Test: Bandwidth - Threads: 8
Mbits/sec > Higher Is Better
ConnectX: Windows 2019 ....... 9143 |==========================================
ConnectX: Ubuntu 18.04 ....... 8803 |========================================
ConnectX: Debian 9.6 ......... 8836 |=========================================
ConnectX: Ubuntu 18.10 ....... 7095 |=================================
ConnectX: Ubuntu 18.10 Tuned . 7133 |=================================
ConnectX: Scientific 7 ....... 8904 |=========================================
ConnectX: Scientific 7 Tuned . 8840 |=========================================


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: UDP - Test: Bandwidth - Threads: 8
Mbits/sec > Higher Is Better
ConnectX: Windows 2019 ....... 9567 |==========================================
ConnectX: Ubuntu 18.04 ....... 9589 |==========================================
ConnectX: Debian 9.6 ......... 9598 |==========================================
ConnectX: Ubuntu 18.10 ....... 9599 |==========================================
ConnectX: Ubuntu 18.10 Tuned . 9600 |==========================================
ConnectX: Scientific 7 ....... 9455 |=========================================
ConnectX: Scientific 7 Tuned . 9599 |==========================================


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: HTTP - Test: Bandwidth - Threads: 1
Mbits/sec > Higher Is Better
ConnectX: Windows 2019 ....... 587 |==================================
ConnectX: Ubuntu 18.04 ....... 581 |=================================
ConnectX: Debian 9.6 ......... 573 |=================================
ConnectX: Ubuntu 18.10 ....... 470 |===========================
ConnectX: Ubuntu 18.10 Tuned . 602 |===================================
ConnectX: Scientific 7 ....... 539 |===============================
ConnectX: Scientific 7 Tuned . 746 |===========================================


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: TCP - Test: Bandwidth - Threads: 64
Mbits/sec > Higher Is Better
ConnectX: Windows 2019 ....... 7795 |==========================================
ConnectX: Ubuntu 18.04 ....... 7524 |=========================================
ConnectX: Debian 9.6 ......... 7784 |==========================================
ConnectX: Ubuntu 18.10 ....... 7110 |======================================
ConnectX: Ubuntu 18.10 Tuned . 7157 |=======================================
ConnectX: Scientific 7 ....... 7430 |========================================
ConnectX: Scientific 7 Tuned . 7458 |========================================


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: TCP - Test: Connections/s - Threads: 1
Connections/sec > Higher Is Better
ConnectX: Windows 2019 ....... 5350 |=======================
ConnectX: Ubuntu 18.04 ....... 4123 |=================
ConnectX: Debian 9.6 ......... 9925 |==========================================
ConnectX: Ubuntu 18.10 ....... 6073 |==========================
ConnectX: Ubuntu 18.10 Tuned . 6960 |=============================
ConnectX: Scientific 7 ....... 2893 |============
ConnectX: Scientific 7 Tuned . 8763 |=====================================


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: TCP - Test: Connections/s - Threads: 8
Connections/sec > Higher Is Better
ConnectX: Windows 2019 ....... 44917 |=================================
ConnectX: Ubuntu 18.04 ....... 52903 |=======================================
ConnectX: Debian 9.6 ......... 43853 |================================
ConnectX: Ubuntu 18.10 ....... 55902 |=========================================
ConnectX: Ubuntu 18.10 Tuned . 39769 |=============================
ConnectX: Scientific 7 ....... 37590 |============================
ConnectX: Scientific 7 Tuned . 40478 |==============================