10GbE Linux Networking Performance Tests
Tests by Michael Larabel for a future article of 10GbE Linux network benchmarks.


Server 1: CentOS 7: 

	Processor: 2 x AMD EPYC 7601 32-Core (64 Cores / 128 Threads), Motherboard: Dell 02MJ3T (1.2.5 BIOS), Chipset: AMD Family 17h, Memory: 516096MB, Disk: 20 x 500GB Samsung SSD 860 + 120GB SSDSCKJB120G7R, Graphics: llvmpipe 504GB, Monitor: VE228, Network: 2 x Broadcom BCM57416 NetXtreme-E Dual-Media 10G RDMA + 2 x Broadcom NetXtreme BCM5720 PCIe

	OS: CentOS Linux 7, Kernel: 3.10.0-957.1.3.el7.x86_64 (x86_64), Desktop: GNOME Shell 3.28.3, Display Server: X Server 1.20.1, Display Driver: modesetting 1.20.1, OpenGL: 2.1 Mesa 18.0.5 (LLVM 6.0 128 bits), Compiler: GCC 4.8.5 20150623, File-System: xfs, Screen Resolution: 1600x1200

Server 1: Debian 9.6: 

	Processor: 2 x AMD EPYC 7601 32-Core (64 Cores / 128 Threads), Motherboard: Dell 02MJ3T (1.2.5 BIOS), Chipset: AMD Device 1450, Memory: 16 x 32 GB DDR4-2400MHz 36ASF4G72PZ-2G6D2, Disk: 20 x 500GB Samsung SSD 860 + 120GB SSDSCKJB120G7R, Graphics: mgadrmfb, Monitor: VE228, Network: 2 x Broadcom BCM57416 NetXtreme-E 10GBase-T RDMA + 2 x Broadcom NetXtreme BCM5720 PCIe

	OS: Debian 9.6, Kernel: 4.9.0-8-amd64 (x86_64), Desktop: GNOME Shell 3.22.3, Display Server: X Server 1.19.2, Compiler: GCC 6.3.0 20170516, File-System: ext4, Screen Resolution: 1600x1200

Server 1: Clear Linux: 

	Processor: 2 x AMD EPYC 7601 32-Core (64 Cores / 128 Threads), Motherboard: Dell 02MJ3T (1.2.5 BIOS), Chipset: AMD Family 17h, Memory: 516096MB, Disk: 20 x 500GB Samsung SSD 860 + 120GB SSDSCKJB120G7R, Graphics: llvmpipe 504GB, Monitor: VE228, Network: 2 x Broadcom BCM57416 NetXtreme-E Dual-Media 10G RDMA + 2 x Broadcom NetXtreme BCM5720 2-port PCIe

	OS: Clear Linux OS 27180, Kernel: 4.19.14-682.native (x86_64), Desktop: GNOME Shell 3.30.2, Display Server: X Server 1.20.3, Display Driver: modesetting 1.20.3, OpenGL: 3.3 Mesa 19.0.0-devel (LLVM 7.0 128 bits), Compiler: GCC 8.2.1 20180502 + Clang 7.0.1 + LLVM 7.0.1, File-System: ext4, Screen Resolution: 1600x1200

Server 1: Fedora Server 29: 

	Processor: 2 x AMD EPYC 7601 32-Core (64 Cores / 128 Threads), Motherboard: Dell 02MJ3T (1.2.5 BIOS), Chipset: AMD Family 17h, Memory: 516096MB, Disk: 120GB SSDSCKJB120G7R + 20 x 500GB Samsung SSD 860, Graphics: Matrox G200eW3, Monitor: VE228, Network: 2 x Broadcom BCM57416 NetXtreme-E Dual-Media 10G RDMA + 2 x Broadcom NetXtreme BCM5720 2-port PCIe

	OS: Fedora 29, Kernel: 4.19.13-300.fc29.x86_64 (x86_64), Compiler: GCC 8.2.1 20181215, File-System: xfs, Screen Resolution: 1600x1200

Server 2: Fedora Server 29: 

	Processor: 2 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads), Motherboard: TYAN S7106 (V1.02.B20 BIOS), Chipset: Intel Sky Lake-E DMI3 Registers, Memory: 96256MB, Disk: Samsung SSD 970 EVO 250GB, Graphics: ASPEED Family, Monitor: VE228, Network: 2 x Intel I210 + 2 x QLogic cLOM8214 1/10GbE

	OS: Fedora 29, Kernel: 4.19.13-300.fc29.x86_64 (x86_64), Compiler: GCC 8.2.1 20181215, File-System: xfs, Screen Resolution: 1920x1080

Server 2: Debian 9.6: 

	Processor: 2 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads), Motherboard: TYAN S7106 (V1.02.B20 BIOS), Chipset: Intel Device 2020, Memory: 12 x 8192 MB DDR4-2666MHz Micron 9ASF1G72PZ-2G6B1, Disk: Samsung SSD 970 EVO 250GB, Graphics: ASPEED Family, Network: 2 x Intel I210 + 2 x QLogic cLOM8214 1/10GbE

	OS: Debian 9.6, Kernel: 4.9.0-8-amd64 (x86_64), Desktop: GNOME Shell 3.22.3, Display Server: X Server 1.19.2, Compiler: GCC 6.3.0 20170516, File-System: ext4, Screen Resolution: 1024x768

Server 2: CentOS 7: 

	Processor: 2 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads), Motherboard: TYAN S7106 (V1.02.B20 BIOS), Chipset: Intel Sky Lake-E DMI3 Registers, Memory: 95232MB, Disk: Samsung SSD 970 EVO 250GB, Graphics: llvmpipe 93GB, Monitor: VE228, Network: 2 x Intel I210 + 2 x QLogic cLOM8214 1/10GbE

	OS: CentOS Linux 7, Kernel: 3.10.0-957.1.3.el7.x86_64 (x86_64), Desktop: GNOME Shell 3.28.3, Display Server: X Server 1.20.1, Display Driver: modesetting 1.20.1, OpenGL: 2.1 Mesa 18.0.5 (LLVM 6.0 256 bits), Compiler: GCC 4.8.5 20150623, File-System: xfs, Screen Resolution: 1920x1080

Server 2: Clear Linux: 

	Processor: 2 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads), Motherboard: TYAN S7106 (V1.02.B20 BIOS), Chipset: Intel Sky Lake-E DMI3 Registers, Memory: 96256MB, Disk: Samsung SSD 970 EVO 250GB, Graphics: llvmpipe 93GB, Monitor: VE228, Network: 2 x Intel I210 + 2 x QLogic cLOM8214 1/10GbE

	OS: Clear Linux OS 27230, Kernel: 4.20.2-683.native (x86_64), Desktop: GNOME Shell 3.30.2, Display Server: X Server 1.20.3, Display Driver: modesetting 1.20.3, OpenGL: 3.3 Mesa 19.0.0-devel (LLVM 7.0 256 bits), Compiler: GCC 8.2.1 20180502 + Clang 7.0.1 + LLVM 7.0.1, File-System: ext4, Screen Resolution: 1920x1080


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: TCP - Test: Latency - Threads: 1
Microseconds < Lower Is Better
Server 1: CentOS 7 ......... 59.45  |=======================
Server 1: Debian 9.6 ....... 52.16  |====================
Server 1: Clear Linux ...... 61.98  |========================
Server 1: Fedora Server 29 . 65.93  |==========================
Server 2: Fedora Server 29 . 104.71 |=========================================
Server 2: Debian 9.6 ....... 80.18  |===============================
Server 2: CentOS 7 ......... 107.62 |==========================================
Server 2: Clear Linux ...... 59.12  |=======================


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: TCP - Test: Latency - Threads: 2
Microseconds < Lower Is Better
Server 1: CentOS 7 ......... 58.43  |=======================
Server 1: Debian 9.6 ....... 56.40  |======================
Server 1: Clear Linux ...... 64.28  |=========================
Server 1: Fedora Server 29 . 64.26  |=========================
Server 2: Fedora Server 29 . 104.62 |=========================================
Server 2: Debian 9.6 ....... 73.02  |=============================
Server 2: CentOS 7 ......... 106.03 |==========================================
Server 2: Clear Linux ...... 60.37  |========================


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: TCP - Test: Latency - Threads: 8
Microseconds < Lower Is Better
Server 1: CentOS 7 ......... 57.35  |=======================
Server 1: Debian 9.6 ....... 53.89  |======================
Server 1: Clear Linux ...... 63.58  |=========================
Server 1: Fedora Server 29 . 59.62  |========================
Server 2: Fedora Server 29 . 102.47 |=========================================
Server 2: Debian 9.6 ....... 69.72  |============================
Server 2: CentOS 7 ......... 104.82 |==========================================
Server 2: Clear Linux ...... 61.03  |========================


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: TCP - Test: Latency - Threads: 16
Microseconds < Lower Is Better
Server 1: CentOS 7 ......... 59.83  |=======================
Server 1: Debian 9.6 ....... 52.67  |====================
Server 1: Clear Linux ...... 65.81  |==========================
Server 1: Fedora Server 29 . 64.99  |=========================
Server 2: Fedora Server 29 . 102.13 |========================================
Server 2: Debian 9.6 ....... 71.67  |============================
Server 2: CentOS 7 ......... 108.15 |==========================================
Server 2: Clear Linux ...... 60.16  |=======================


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: TCP - Test: Latency - Threads: 32
Microseconds < Lower Is Better
Server 1: CentOS 7 ......... 59.44  |=======================
Server 1: Debian 9.6 ....... 53.80  |=====================
Server 1: Clear Linux ...... 61.77  |========================
Server 1: Fedora Server 29 . 63.83  |=========================
Server 2: Fedora Server 29 . 106.07 |==========================================
Server 2: Debian 9.6 ....... 70.78  |============================
Server 2: CentOS 7 ......... 106.54 |==========================================
Server 2: Clear Linux ...... 61.68  |========================


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: TCP - Test: Latency - Threads: 64
Microseconds < Lower Is Better
Server 1: CentOS 7 ......... 59.88  |========================
Server 1: Debian 9.6 ....... 54.29  |=====================
Server 1: Clear Linux ...... 63.09  |=========================
Server 1: Fedora Server 29 . 60.28  |========================
Server 2: Fedora Server 29 . 104.18 |=========================================
Server 2: Debian 9.6 ....... 70.62  |============================
Server 2: CentOS 7 ......... 106.56 |==========================================
Server 2: Clear Linux ...... 61.81  |========================


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: HTTP - Test: Bandwidth - Threads: 1
Mbits/sec > Higher Is Better
Server 1: CentOS 7 ......... 445 |===================================
Server 1: Debian 9.6 ....... 580 |=============================================
Server 1: Clear Linux ...... 533 |=========================================
Server 1: Fedora Server 29 . 580 |=============================================
Server 2: Fedora Server 29 . 340 |==========================
Server 2: Debian 9.6 ....... 412 |================================
Server 2: CentOS 7 ......... 336 |==========================
Server 2: Clear Linux ...... 465 |====================================


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: HTTP - Test: Bandwidth - Threads: 2
Mbits/sec > Higher Is Better
Server 1: CentOS 7 ......... 803 |=====================================
Server 1: Debian 9.6 ....... 934 |===========================================
Server 1: Clear Linux ...... 944 |===========================================
Server 1: Fedora Server 29 . 983 |=============================================
Server 2: Fedora Server 29 . 628 |=============================
Server 2: Debian 9.6 ....... 762 |===================================
Server 2: CentOS 7 ......... 629 |=============================
Server 2: Clear Linux ...... 847 |=======================================


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: TCP - Test: Connections/s - Threads: 1
Connections/sec > Higher Is Better
Server 1: CentOS 7 ......... 5730  |===================
Server 1: Debian 9.6 ....... 12870 |===========================================
Server 1: Clear Linux ...... 8233  |============================
Server 1: Fedora Server 29 . 8877  |==============================
Server 2: Fedora Server 29 . 5077  |=================
Server 2: Debian 9.6 ....... 9283  |===============================
Server 2: CentOS 7 ......... 3617  |============
Server 2: Clear Linux ...... 10427 |===================================


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: TCP - Test: Connections/s - Threads: 2
Connections/sec > Higher Is Better
Server 1: CentOS 7 ......... 14280 |=========================
Server 1: Debian 9.6 ....... 24756 |===========================================
Server 1: Clear Linux ...... 15433 |===========================
Server 1: Fedora Server 29 . 17507 |==============================
Server 2: Fedora Server 29 . 13017 |=======================
Server 2: Debian 9.6 ....... 19760 |==================================
Server 2: CentOS 7 ......... 8470  |===============
Server 2: Clear Linux ...... 22193 |=======================================


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: TCP - Test: Connections/s - Threads: 8
Connections/sec > Higher Is Better
Server 1: CentOS 7 ......... 44563 |===================================
Server 1: Debian 9.6 ....... 54250 |===========================================
Server 1: Clear Linux ...... 51647 |=========================================
Server 1: Fedora Server 29 . 48467 |======================================
Server 2: Fedora Server 29 . 39573 |===============================
Server 2: Debian 9.6 ....... 52497 |==========================================
Server 2: CentOS 7 ......... 43750 |===================================
Server 2: Clear Linux ...... 47419 |======================================


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: TCP - Test: Connections/s - Threads: 16
Connections/sec > Higher Is Better
Server 1: CentOS 7 ......... 37746  |============
Server 1: Debian 9.6 ....... 45344  |===============
Server 1: Clear Linux ...... 46708  |===============
Server 1: Fedora Server 29 . 127891 |==========================================
Server 2: Fedora Server 29 . 48949  |================
Server 2: Debian 9.6 ....... 45338  |===============
Server 2: CentOS 7 ......... 33191  |===========
Server 2: Clear Linux ...... 31373  |==========


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: TCP - Test: Connections/s - Threads: 32
Connections/sec > Higher Is Better
Server 1: CentOS 7 ......... 24751 |=======================
Server 1: Debian 9.6 ....... 40109 |=====================================
Server 1: Clear Linux ...... 46773 |===========================================
Server 1: Fedora Server 29 . 45698 |==========================================
Server 2: Fedora Server 29 . 46620 |===========================================
Server 2: Debian 9.6 ....... 36452 |==================================
Server 2: CentOS 7 ......... 27607 |=========================
Server 2: Clear Linux ...... 41296 |======================================


Ethr 2019-01-02
Server Address: 192.168.1.83 - Protocol: TCP - Test: Connections/s - Threads: 64
Connections/sec > Higher Is Better
Server 1: CentOS 7 ......... 36107 |==================================
Server 1: Debian 9.6 ....... 45014 |===========================================
Server 1: Clear Linux ...... 36690 |===================================
Server 1: Fedora Server 29 . 35077 |==================================
Server 2: Fedora Server 29 . 44541 |===========================================
Server 2: Debian 9.6 ....... 44914 |===========================================
Server 2: CentOS 7 ......... 24411 |=======================
Server 2: Clear Linux ...... 37602 |====================================


iPerf 3.1.3
Server Address: 192.168.1.83 - Server Port: 5201 - Duration: 60 Seconds - Test: UDP - 1000Mbit Objective - Parallel: 5
Mbits/sec > Higher Is Better
Server 1: CentOS 7 ......... 4744 |==========================================
Server 1: Debian 9.6 ....... 4998 |============================================
Server 1: Clear Linux ...... 4862 |===========================================
Server 1: Fedora Server 29 . 2641 |=======================
Server 2: Fedora Server 29 . 4996 |============================================
Server 2: Debian 9.6 ....... 4994 |============================================
Server 2: CentOS 7 ......... 4995 |============================================
Server 2: Clear Linux ...... 4858 |===========================================


iPerf 3.1.3
Server Address: 192.168.1.83 - Server Port: 5201 - Duration: 60 Seconds - Test: UDP - 1000Mbit Objective - Parallel: 10
Mbits/sec > Higher Is Better
Server 1: CentOS 7 ......... 4627 |==========================
Server 1: Debian 9.6 ....... 5484 |==============================
Server 1: Clear Linux ...... 5246 |=============================
Server 1: Fedora Server 29 . 2811 |================
Server 2: Fedora Server 29 . 6863 |======================================
Server 2: Debian 9.6 ....... 7633 |==========================================
Server 2: CentOS 7 ......... 7916 |============================================
Server 2: Clear Linux ...... 5449 |==============================


iPerf 3.1.3
Server Address: 192.168.1.83 - Server Port: 5201 - Duration: 60 Seconds - Test: UDP - 1000Mbit Objective - Parallel: 20
Mbits/sec > Higher Is Better
Server 1: CentOS 7 ......... 4911 |==========================
Server 1: Debian 9.6 ....... 4818 |==========================
Server 1: Clear Linux ...... 4401 |========================
Server 1: Fedora Server 29 . 2215 |============
Server 2: Fedora Server 29 . 4977 |===========================
Server 2: Debian 9.6 ....... 5112 |===========================
Server 2: CentOS 7 ......... 8228 |============================================
Server 2: Clear Linux ...... 4809 |==========================


iPerf 3.1.3
Server Address: 192.168.1.83 - Server Port: 5201 - Duration: 60 Seconds - Test: TCP - Parallel: 1
Mbits/sec > Higher Is Better
Server 1: CentOS 7 ......... 9182 |============================================
Server 1: Debian 9.6 ....... 9031 |===========================================
Server 1: Clear Linux ...... 6978 |=================================
Server 1: Fedora Server 29 . 9211 |============================================
Server 2: Fedora Server 29 . 7603 |====================================
Server 2: Debian 9.6 ....... 7973 |======================================
Server 2: CentOS 7 ......... 7625 |====================================
Server 2: Clear Linux ...... 6000 |=============================


iPerf 3.1.3
Server Address: 192.168.1.83 - Server Port: 5201 - Duration: 60 Seconds - Test: TCP - Parallel: 5
Mbits/sec > Higher Is Better
Server 1: CentOS 7 ......... 8735 |===========================================
Server 1: Debian 9.6 ....... 8795 |===========================================
Server 1: Clear Linux ...... 7586 |=====================================
Server 1: Fedora Server 29 . 8725 |===========================================
Server 2: Fedora Server 29 . 8907 |============================================
Server 2: Debian 9.6 ....... 8994 |============================================
Server 2: CentOS 7 ......... 8878 |===========================================
Server 2: Clear Linux ...... 7415 |====================================


iPerf 3.1.3
Server Address: 192.168.1.83 - Server Port: 5201 - Duration: 60 Seconds - Test: TCP - Parallel: 20
Mbits/sec > Higher Is Better
Server 1: CentOS 7 ......... 8535 |===========================================
Server 1: Debian 9.6 ....... 7063 |===================================
Server 1: Clear Linux ...... 7844 |=======================================
Server 1: Fedora Server 29 . 8240 |=========================================
Server 2: Fedora Server 29 . 8712 |===========================================
Server 2: Debian 9.6 ....... 8798 |============================================
Server 2: CentOS 7 ......... 8817 |============================================
Server 2: Clear Linux ...... 7893 |=======================================


Nuttcp 8.1.4
Test: TCP Transfer - Default - Server Address: 192.168.1.83
Mbits/sec > Higher Is Better
Server 1: CentOS 7 ......... 9231 |============================================
Server 1: Debian 9.6 ....... 9093 |===========================================
Server 1: Clear Linux ...... 6840 |=================================
Server 1: Fedora Server 29 . 9207 |============================================
Server 2: Fedora Server 29 . 7505 |====================================
Server 2: Debian 9.6 ....... 8034 |======================================
Server 2: CentOS 7 ......... 7562 |====================================
Server 2: Clear Linux ...... 5955 |============================


Netperf 2.7.0
Server: 192.168.1.83 - Test: TCP Request Response - Duration: 60 Seconds
Transaction Rate Per Second > Higher Is Better
Server 1: CentOS 7 ......... 2068 |===========
Server 1: Debian 9.6 ....... 3014 |================
Server 1: Clear Linux ...... 3938 |=====================
Server 1: Fedora Server 29 . 3090 |=================
Server 2: Fedora Server 29 . 2252 |============
Server 2: Debian 9.6 ....... 5449 |==============================
Server 2: CentOS 7 ......... 1331 |=======
Server 2: Clear Linux ...... 8110 |============================================


Netperf 2.7.0
Server: 192.168.1.83 - Test: UDP Request Response - Duration: 60 Seconds
Transaction Rate Per Second > Higher Is Better
Server 1: CentOS 7 ......... 1259 |=======
Server 1: Debian 9.6 ....... 3634 |===================
Server 1: Clear Linux ...... 1683 |=========
Server 1: Fedora Server 29 . 3698 |====================
Server 2: Fedora Server 29 . 1929 |==========
Server 2: Debian 9.6 ....... 4896 |==========================
Server 2: CentOS 7 ......... 1689 |=========
Server 2: Clear Linux ...... 8218 |============================================


Netperf 2.7.0
Server: 192.168.1.83 - Test: TCP Stream - Client To Server - Duration: 60 Seconds
Megabits/sec Throughput > Higher Is Better
Server 1: CentOS 7 ......... 677 |======================================
Server 1: Debian 9.6 ....... 601 |=================================
Server 1: Clear Linux ...... 580 |================================
Server 1: Fedora Server 29 . 606 |==================================
Server 2: Fedora Server 29 . 772 |===========================================
Server 2: Debian 9.6 ....... 685 |======================================
Server 2: CentOS 7 ......... 667 |=====================================
Server 2: Clear Linux ...... 812 |=============================================


Netperf 2.7.0
Server: 192.168.1.83 - Test: TCP Stream - Server To Client - Duration: 60 Seconds
Megabits/sec Throughput > Higher Is Better
Server 1: CentOS 7 ......... 242.00 |============
Server 1: Debian 9.6 ....... 78.47  |====
Server 1: Clear Linux ...... 860.22 |==========================================
Server 1: Fedora Server 29 . 196.83 |==========
Server 2: Fedora Server 29 . 565.42 |============================
Server 2: Debian 9.6 ....... 797.45 |=======================================
Server 2: CentOS 7 ......... 638.43 |===============================
Server 2: Clear Linux ...... 91.01  |====