10GbE Linux vs. Windows vs. BSD Network Benchmarks

10GbE Linux network performance benchmarks by Michael Larabel for a future article on Phoronix.com.

HTML result view exported from: https://openbenchmarking.org/result/1901259-SP-NETWORK1072.

ProcessorMotherboardMemoryDiskGraphicsNetworkChipsetMonitorOSKernelDisplay DriverFile-SystemScreen ResolutionCompilerDesktopDisplay ServerOpenGLConnectX Windows 2019 FreeBSD 12 Ubuntu 18.04 Debian 9.6 Ubuntu 18.10 Ubuntu 18.10 Tuned Scientific 7 Scientific 7 Tuned2 x Intel Xeon Gold 6138 @ 2.00GHz (40 Cores / 80 Threads)TYAN S710612 x 8192 MB 2666MHz Micron 9ASF1G72PZ-2G6B1233GB Samsung SSD 970 EVO 250GBMicrosoft Basic DisplayIntel I210 Gigabit Connection + Intel I210 Gigabit Connection #2Microsoft Windows Server 2019 Essentials Build 1776310.0 (x86_64)10.0.17763.1NTFS1024x768Intel Xeon Gold 6138 @ 2.00GHz (80 Cores)TYAN B7106G24EV4HRIntel Sky Lake-E DMI3 Registers97024MBGeneric NVMe DeviceASPEED FamilyFreeBSD12.0-RELEASE (x86_64)Clang 6.0.1 (SVN 335540)zfs800x6002 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads)TYAN S7106 (V1.02.B20 BIOS)96256MBSamsung SSD 970 EVO 250GBVE2282 x Intel I210 + 2 x QLogic cLOM8214 1/10GbE + Mellanox MT26448Ubuntu 18.044.15.0-43-generic (x86_64)GNOME Shell 3.28.3X Server 1.19.6modesetting 1.19.63.3 Mesa 18.0.5 (LLVM 6.0 256 bits)GCC 7.3.0ext41920x1080Intel Device 202012 x 8192 MB DDR4-2666MHz Micron 9ASF1G72PZ-2G6B1LLVMpipeDebian 9.64.9.0-8-amd64 (x86_64)GNOME Shell 3.22.3X Server 1.19.23.3 Mesa 13.0.6 Gallium 0.4 (LLVM 3.9 256 bits)GCC 6.3.0 201705161024x768Intel Sky Lake-E DMI3 Registers96256MBllvmpipe 93GBVE228Ubuntu 18.104.18.0-13-generic (x86_64)GNOME Shell 3.30.1X Server 1.20.1modesetting 1.20.13.3 Mesa 18.2.2 (LLVM 7.0 256 bits)GCC 8.2.01920x108012 x 8192 MB DDR4-2666MT/s Micron 9ASF1G72PZ-2G6B1Scientific 7.63.10.0-957.1.3.el7.x86_64 (x86_64)GNOME Shell 3.28.32.1 Mesa 18.0.5 (LLVM 6.0 256 bits)GCC 4.8.5 20150623xfsOpenBenchmarking.orgSecurity Details- ConnectX: FreeBSD 12: KPTI- ConnectX: Ubuntu 18.04: KPTI + __user pointer sanitization + Full generic retpoline IBPB IBRS_FW + SSB disabled via prctl and seccomp + PTE Inversion; VMX: conditional cache flushes SMT vulnerable- ConnectX: Debian 9.6: KPTI + __user pointer sanitization + Full generic retpoline IBPB IBRS_FW + SSB disabled via prctl and seccomp + PTE Inversion; VMX: conditional cache flushes SMT vulnerable- ConnectX: Ubuntu 18.10: KPTI + __user pointer sanitization + Full generic retpoline IBPB IBRS_FW + SSB disabled via prctl and seccomp + PTE Inversion; VMX: conditional cache flushes SMT vulnerable- ConnectX: Ubuntu 18.10 Tuned: KPTI + __user pointer sanitization + Full generic retpoline IBPB IBRS_FW + SSB disabled via prctl and seccomp + PTE Inversion; VMX: conditional cache flushes SMT vulnerable- ConnectX: Scientific 7: SELinux + KPTI + Load fences __user pointer sanitization + IBRS (kernel) + SSB disabled via prctl and seccomp + PTE Inversion; VMX: SMT vulnerable L1D conditional cache flushes- ConnectX: Scientific 7 Tuned: SELinux + KPTI + Load fences __user pointer sanitization + IBRS (kernel) + SSB disabled via prctl and seccomp + PTE Inversion; VMX: SMT vulnerable L1D conditional cache flushesCompiler Details- ConnectX: Ubuntu 18.04: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic --without-cuda-driver -v - ConnectX: Debian 9.6: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - ConnectX: Ubuntu 18.10: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic --without-cuda-driver -v - ConnectX: Ubuntu 18.10 Tuned: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic --without-cuda-driver -v - ConnectX: Scientific 7: --build=x86_64-redhat-linux --disable-libgcj --disable-libunwind-exceptions --enable-__cxa_atexit --enable-bootstrap --enable-checking=release --enable-gnu-indirect-function --enable-gnu-unique-object --enable-initfini-array --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-shared --enable-threads=posix --mandir=/usr/share/man --with-arch_32=x86-64 --with-linker-hash-style=gnu --with-tune=generic- ConnectX: Scientific 7 Tuned: --build=x86_64-redhat-linux --disable-libgcj --disable-libunwind-exceptions --enable-__cxa_atexit --enable-bootstrap --enable-checking=release --enable-gnu-indirect-function --enable-gnu-unique-object --enable-initfini-array --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-shared --enable-threads=posix --mandir=/usr/share/man --with-arch_32=x86-64 --with-linker-hash-style=gnu --with-tune=genericProcessor Details- ConnectX: Ubuntu 18.04: Scaling Governor: intel_pstate powersave- ConnectX: Debian 9.6: Scaling Governor: intel_pstate powersave- ConnectX: Ubuntu 18.10: Scaling Governor: intel_pstate powersave- ConnectX: Ubuntu 18.10 Tuned: Scaling Governor: intel_pstate performance- ConnectX: Scientific 7: Scaling Governor: intel_pstate powersave- ConnectX: Scientific 7 Tuned: Scaling Governor: intel_pstate performance

iperf: 192.168.1.83 - 5201 - 60 Seconds - TCP - 1iperf: 192.168.1.83 - 5201 - 60 Seconds - TCP - 5iperf: 192.168.1.83 - 5201 - 60 Seconds - TCP - 20nuttcp: TCP Transfer - Default - 192.168.1.83ethr: 192.168.1.83 - TCP - Latency - 1ethr: 192.168.1.83 - TCP - Latency - 8ethr: 192.168.1.83 - TCP - Latency - 64ethr: 192.168.1.83 - TCP - Bandwidth - 8ethr: 192.168.1.83 - UDP - Bandwidth - 8ethr: 192.168.1.83 - HTTP - Bandwidth - 1ethr: 192.168.1.83 - TCP - Bandwidth - 64ethr: 192.168.1.83 - TCP - Connections/s - 1ethr: 192.168.1.83 - TCP - Connections/s - 8ConnectX Windows 2019 FreeBSD 12 Ubuntu 18.04 Debian 9.6 Ubuntu 18.10 Ubuntu 18.10 Tuned Scientific 7 Scientific 7 Tuned567292529098942064.1064.9464.489143956758777955350449178640887986268909917888628713915656.3457.9452.45880395895817524412352903916588748750917752.6752.7552.80883695985737784992543853798269406701799756.8260.6057.62709595994707110607355902784670146709790947.0844.6246.19713396006027157696039769917588108726925171.5671.2471.45890494555397430289337590917389358708925747.1246.6245.48884095997467458876340478OpenBenchmarking.org

iPerf

Server Address: 192.168.1.83 - Server Port: 5201 - Duration: 60 Seconds - Test: TCP - Parallel: 1

OpenBenchmarking.orgMbits/sec, More Is BetteriPerf 3.1.3Server Address: 192.168.1.83 - Server Port: 5201 - Duration: 60 Seconds - Test: TCP - Parallel: 1Windows 2019FreeBSD 12Ubuntu 18.04Debian 9.6Ubuntu 18.10Ubuntu 18.10 TunedScientific 7Scientific 7 Tuned2K4K6K8K10KSE +/- 39.67, N = 3SE +/- 117.46, N = 3SE +/- 23.36, N = 3SE +/- 4.36, N = 3SE +/- 106.96, N = 3SE +/- 27.43, N = 3SE +/- 15.34, N = 3SE +/- 1.53, N = 356728640917891657982784691759173clanggccgccgccgccgccgcc

iPerf

Server Address: 192.168.1.83 - Server Port: 5201 - Duration: 60 Seconds - Test: TCP - Parallel: 5

OpenBenchmarking.orgMbits/sec, More Is BetteriPerf 3.1.3Server Address: 192.168.1.83 - Server Port: 5201 - Duration: 60 Seconds - Test: TCP - Parallel: 5Windows 2019FreeBSD 12Ubuntu 18.04Debian 9.6Ubuntu 18.10Ubuntu 18.10 TunedScientific 7Scientific 7 Tuned2K4K6K8K10KSE +/- 49.49, N = 3SE +/- 79.51, N = 3SE +/- 76.70, N = 3SE +/- 40.40, N = 3SE +/- 59.21, N = 3SE +/- 48.09, N = 3SE +/- 83.30, N = 3SE +/- 37.69, N = 392528879886288746940701488108935clanggccgccgccgccgccgcc

iPerf

Server Address: 192.168.1.83 - Server Port: 5201 - Duration: 60 Seconds - Test: TCP - Parallel: 20

OpenBenchmarking.orgMbits/sec, More Is BetteriPerf 3.1.3Server Address: 192.168.1.83 - Server Port: 5201 - Duration: 60 Seconds - Test: TCP - Parallel: 20Windows 2019FreeBSD 12Ubuntu 18.04Debian 9.6Ubuntu 18.10Ubuntu 18.10 TunedScientific 7Scientific 7 Tuned2K4K6K8K10KSE +/- 17.52, N = 3SE +/- 16.51, N = 3SE +/- 21.66, N = 3SE +/- 19.72, N = 3SE +/- 21.59, N = 3SE +/- 11.26, N = 3SE +/- 17.89, N = 3SE +/- 17.78, N = 390988626871387506701670987268708clanggccgccgccgccgccgcc

Nuttcp

Test: TCP Transfer - Default - Server Address: 192.168.1.83

OpenBenchmarking.orgMbits/sec, More Is BetterNuttcp 8.1.4Test: TCP Transfer - Default - Server Address: 192.168.1.83Windows 2019FreeBSD 12Ubuntu 18.04Debian 9.6Ubuntu 18.10Ubuntu 18.10 TunedScientific 7Scientific 7 Tuned2K4K6K8K10KSE +/- 56.96, N = 3SE +/- 36.45, N = 3SE +/- 11.16, N = 3SE +/- 9.90, N = 3SE +/- 46.11, N = 3SE +/- 36.31, N = 3SE +/- 4.19, N = 3SE +/- 6.98, N = 394208909915691777997790992519257clanggccgccgccgccgccgcc

Ethr

Server Address: 192.168.1.83 - Protocol: TCP - Test: Latency - Threads: 1

OpenBenchmarking.orgMicroseconds, Fewer Is BetterEthr 2019-01-02Server Address: 192.168.1.83 - Protocol: TCP - Test: Latency - Threads: 1Windows 2019Ubuntu 18.04Debian 9.6Ubuntu 18.10Ubuntu 18.10 TunedScientific 7Scientific 7 Tuned1632486480SE +/- 1.36, N = 9SE +/- 1.07, N = 12SE +/- 0.62, N = 12SE +/- 0.31, N = 3SE +/- 1.30, N = 9SE +/- 0.74, N = 12SE +/- 0.57, N = 864.1056.3452.6756.8247.0871.5647.12MAX: 999.6MIN: 46.56 / MAX: 103.3MIN: 42.98 / MAX: 91.19MIN: 48.28 / MAX: 98.87MIN: 36.01 / MAX: 153.68MIN: 56.02 / MAX: 181.31MIN: 40.36 / MAX: 145.47

Ethr

Server Address: 192.168.1.83 - Protocol: TCP - Test: Latency - Threads: 8

OpenBenchmarking.orgMicroseconds, Fewer Is BetterEthr 2019-01-02Server Address: 192.168.1.83 - Protocol: TCP - Test: Latency - Threads: 8Windows 2019Ubuntu 18.04Debian 9.6Ubuntu 18.10Ubuntu 18.10 TunedScientific 7Scientific 7 Tuned1632486480SE +/- 1.58, N = 12SE +/- 1.22, N = 12SE +/- 0.71, N = 12SE +/- 0.90, N = 3SE +/- 1.00, N = 12SE +/- 1.25, N = 3SE +/- 0.45, N = 364.9457.9452.7560.6044.6271.2446.62MAX: 998.9MIN: 42.98 / MAX: 101.92MIN: 44.17 / MAX: 85.95MIN: 50.65 / MAX: 115.6MIN: 34.03 / MAX: 158.03MIN: 56.85 / MAX: 181.46MIN: 40.43 / MAX: 200.08

Ethr

Server Address: 192.168.1.83 - Protocol: TCP - Test: Latency - Threads: 64

OpenBenchmarking.orgMicroseconds, Fewer Is BetterEthr 2019-01-02Server Address: 192.168.1.83 - Protocol: TCP - Test: Latency - Threads: 64Windows 2019Ubuntu 18.04Debian 9.6Ubuntu 18.10Ubuntu 18.10 TunedScientific 7Scientific 7 Tuned1632486480SE +/- 1.07, N = 12SE +/- 0.17, N = 3SE +/- 0.73, N = 3SE +/- 0.59, N = 3SE +/- 0.92, N = 12SE +/- 1.04, N = 3SE +/- 0.60, N = 664.4852.4552.8057.6246.1971.4545.48MAX: 998.7MIN: 44.78 / MAX: 87.58MIN: 44.43 / MAX: 85.1MIN: 49.44 / MAX: 102.49MIN: 35.38 / MAX: 148.13MIN: 57.94 / MAX: 180.34MIN: 38.82 / MAX: 180.07

Ethr

Server Address: 192.168.1.83 - Protocol: TCP - Test: Bandwidth - Threads: 8

OpenBenchmarking.orgMbits/sec, More Is BetterEthr 2019-01-02Server Address: 192.168.1.83 - Protocol: TCP - Test: Bandwidth - Threads: 8Windows 2019Ubuntu 18.04Debian 9.6Ubuntu 18.10Ubuntu 18.10 TunedScientific 7Scientific 7 Tuned2K4K6K8K10KSE +/- 48.30, N = 3SE +/- 37.44, N = 3SE +/- 10.26, N = 3SE +/- 26.03, N = 3SE +/- 26.00, N = 3SE +/- 4.02, N = 3SE +/- 21.51, N = 39143880388367095713389048840

Ethr

Server Address: 192.168.1.83 - Protocol: UDP - Test: Bandwidth - Threads: 8

OpenBenchmarking.orgMbits/sec, More Is BetterEthr 2019-01-02Server Address: 192.168.1.83 - Protocol: UDP - Test: Bandwidth - Threads: 8Windows 2019Ubuntu 18.04Debian 9.6Ubuntu 18.10Ubuntu 18.10 TunedScientific 7Scientific 7 Tuned2K4K6K8K10KSE +/- 31.01, N = 3SE +/- 2.70, N = 3SE +/- 1.56, N = 3SE +/- 0.77, N = 3SE +/- 122.02, N = 3SE +/- 0.67, N = 39567958995989599960094559599

Ethr

Server Address: 192.168.1.83 - Protocol: HTTP - Test: Bandwidth - Threads: 1

OpenBenchmarking.orgMbits/sec, More Is BetterEthr 2019-01-02Server Address: 192.168.1.83 - Protocol: HTTP - Test: Bandwidth - Threads: 1Windows 2019Ubuntu 18.04Debian 9.6Ubuntu 18.10Ubuntu 18.10 TunedScientific 7Scientific 7 Tuned160320480640800SE +/- 9.83, N = 4SE +/- 3.60, N = 3SE +/- 6.28, N = 10SE +/- 2.13, N = 3SE +/- 2.05, N = 3SE +/- 0.26, N = 3SE +/- 2.53, N = 3587581573470602539746

Ethr

Server Address: 192.168.1.83 - Protocol: TCP - Test: Bandwidth - Threads: 64

OpenBenchmarking.orgMbits/sec, More Is BetterEthr 2019-01-02Server Address: 192.168.1.83 - Protocol: TCP - Test: Bandwidth - Threads: 64Windows 2019Ubuntu 18.04Debian 9.6Ubuntu 18.10Ubuntu 18.10 TunedScientific 7Scientific 7 Tuned2K4K6K8K10KSE +/- 97.74, N = 3SE +/- 32.36, N = 3SE +/- 5.55, N = 3SE +/- 10.13, N = 3SE +/- 2.34, N = 3SE +/- 92.96, N = 3SE +/- 99.26, N = 37795752477847110715774307458

Ethr

Server Address: 192.168.1.83 - Protocol: TCP - Test: Connections/s - Threads: 1

OpenBenchmarking.orgConnections/sec, More Is BetterEthr 2019-01-02Server Address: 192.168.1.83 - Protocol: TCP - Test: Connections/s - Threads: 1Windows 2019Ubuntu 18.04Debian 9.6Ubuntu 18.10Ubuntu 18.10 TunedScientific 7Scientific 7 Tuned2K4K6K8K10KSE +/- 55.08, N = 3SE +/- 63.03, N = 4SE +/- 414.15, N = 12SE +/- 78.39, N = 3SE +/- 20.00, N = 3SE +/- 21.86, N = 3SE +/- 268.40, N = 125350412399256073696028938763

Ethr

Server Address: 192.168.1.83 - Protocol: TCP - Test: Connections/s - Threads: 8

OpenBenchmarking.orgConnections/sec, More Is BetterEthr 2019-01-02Server Address: 192.168.1.83 - Protocol: TCP - Test: Connections/s - Threads: 8Windows 2019Ubuntu 18.04Debian 9.6Ubuntu 18.10Ubuntu 18.10 TunedScientific 7Scientific 7 Tuned12K24K36K48K60KSE +/- 392.82, N = 3SE +/- 84.52, N = 3SE +/- 5762.94, N = 9SE +/- 808.72, N = 5SE +/- 5405.55, N = 12SE +/- 72.11, N = 3SE +/- 2683.50, N = 1244917529034385355902397693759040478


Phoronix Test Suite v10.8.4