10GbE Linux vs. Windows vs. BSD Network Benchmarks
10GbE Linux network performance benchmarks by Michael Larabel for a future article on Phoronix.com.
ConnectX: Windows 2019
Processor: 2 x Intel Xeon Gold 6138 @ 2.00GHz (40 Cores / 80 Threads), Motherboard: TYAN S7106, Memory: 12 x 8192 MB 2666MHz Micron 9ASF1G72PZ-2G6B1, Disk: 233GB Samsung SSD 970 EVO 250GB, Graphics: Microsoft Basic Display, Network: Intel I210 Gigabit Connection + Intel I210 Gigabit Connection #2
OS: Microsoft Windows Server 2019 Essentials Build 17763, Kernel: 10.0 (x86_64), Display Driver: 10.0.17763.1, File-System: NTFS, Screen Resolution: 1024x768
ConnectX: FreeBSD 12
Processor: Intel Xeon Gold 6138 @ 2.00GHz (80 Cores), Motherboard: TYAN B7106G24EV4HR, Chipset: Intel Sky Lake-E DMI3 Registers, Memory: 97024MB, Disk: Generic NVMe Device, Graphics: ASPEED Family
OS: FreeBSD, Kernel: 12.0-RELEASE (x86_64), Compiler: Clang 6.0.1 (SVN 335540), File-System: zfs, Screen Resolution: 800x600
Security Notes: KPTI
ConnectX: Ubuntu 18.04
Processor: 2 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads), Motherboard: TYAN S7106 (V1.02.B20 BIOS), Chipset: Intel Sky Lake-E DMI3 Registers, Memory: 96256MB, Disk: Samsung SSD 970 EVO 250GB, Graphics: ASPEED Family, Monitor: VE228, Network: 2 x Intel I210 + 2 x QLogic cLOM8214 1/10GbE + Mellanox MT26448
OS: Ubuntu 18.04, Kernel: 4.15.0-43-generic (x86_64), Desktop: GNOME Shell 3.28.3, Display Server: X Server 1.19.6, Display Driver: modesetting 1.19.6, OpenGL: 3.3 Mesa 18.0.5 (LLVM 6.0 256 bits), Compiler: GCC 7.3.0, File-System: ext4, Screen Resolution: 1920x1080
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: intel_pstate powersave
Security Notes: KPTI + __user pointer sanitization + Full generic retpoline IBPB IBRS_FW + SSB disabled via prctl and seccomp + PTE Inversion; VMX: conditional cache flushes SMT vulnerable
ConnectX: Debian 9.6
Processor: 2 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads), Motherboard: TYAN S7106 (V1.02.B20 BIOS), Chipset: Intel Device 2020, Memory: 12 x 8192 MB DDR4-2666MHz Micron 9ASF1G72PZ-2G6B1, Disk: Samsung SSD 970 EVO 250GB, Graphics: LLVMpipe, Network: 2 x Intel I210 + 2 x QLogic cLOM8214 1/10GbE + Mellanox MT26448
OS: Debian 9.6, Kernel: 4.9.0-8-amd64 (x86_64), Desktop: GNOME Shell 3.22.3, Display Server: X Server 1.19.2, OpenGL: 3.3 Mesa 13.0.6 Gallium 0.4 (LLVM 3.9 256 bits), Compiler: GCC 6.3.0 20170516, File-System: ext4, Screen Resolution: 1024x768
Compiler Notes: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v
Processor Notes: Scaling Governor: intel_pstate powersave
Security Notes: KPTI + __user pointer sanitization + Full generic retpoline IBPB IBRS_FW + SSB disabled via prctl and seccomp + PTE Inversion; VMX: conditional cache flushes SMT vulnerable
ConnectX: Ubuntu 18.10
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: intel_pstate powersave
Security Notes: KPTI + __user pointer sanitization + Full generic retpoline IBPB IBRS_FW + SSB disabled via prctl and seccomp + PTE Inversion; VMX: conditional cache flushes SMT vulnerable
ConnectX: Ubuntu 18.10 Tuned
Processor: 2 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads), Motherboard: TYAN S7106 (V1.02.B20 BIOS), Chipset: Intel Sky Lake-E DMI3 Registers, Memory: 96256MB, Disk: Samsung SSD 970 EVO 250GB, Graphics: llvmpipe 93GB, Monitor: VE228, Network: 2 x Intel I210 + 2 x QLogic cLOM8214 1/10GbE + Mellanox MT26448
OS: Ubuntu 18.10, Kernel: 4.18.0-13-generic (x86_64), Desktop: GNOME Shell 3.30.1, Display Server: X Server 1.20.1, Display Driver: modesetting 1.20.1, OpenGL: 3.3 Mesa 18.2.2 (LLVM 7.0 256 bits), Compiler: GCC 8.2.0, File-System: ext4, Screen Resolution: 1920x1080
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: intel_pstate performance
Security Notes: KPTI + __user pointer sanitization + Full generic retpoline IBPB IBRS_FW + SSB disabled via prctl and seccomp + PTE Inversion; VMX: conditional cache flushes SMT vulnerable
ConnectX: Scientific 7
Compiler Notes: --build=x86_64-redhat-linux --disable-libgcj --disable-libunwind-exceptions --enable-__cxa_atexit --enable-bootstrap --enable-checking=release --enable-gnu-indirect-function --enable-gnu-unique-object --enable-initfini-array --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-shared --enable-threads=posix --mandir=/usr/share/man --with-arch_32=x86-64 --with-linker-hash-style=gnu --with-tune=generic
Processor Notes: Scaling Governor: intel_pstate powersave
Security Notes: SELinux + KPTI + Load fences __user pointer sanitization + IBRS (kernel) + SSB disabled via prctl and seccomp + PTE Inversion; VMX: SMT vulnerable L1D conditional cache flushes
ConnectX: Scientific 7 Tuned
Processor: 2 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads), Motherboard: TYAN S7106 (V1.02.B20 BIOS), Chipset: Intel Sky Lake-E DMI3 Registers, Memory: 12 x 8192 MB DDR4-2666MT/s Micron 9ASF1G72PZ-2G6B1, Disk: Samsung SSD 970 EVO 250GB, Graphics: llvmpipe 93GB, Monitor: VE228, Network: 2 x Intel I210 + 2 x QLogic cLOM8214 1/10GbE + Mellanox MT26448
OS: Scientific 7.6, Kernel: 3.10.0-957.1.3.el7.x86_64 (x86_64), Desktop: GNOME Shell 3.28.3, Display Server: X Server 1.20.1, Display Driver: modesetting 1.20.1, OpenGL: 2.1 Mesa 18.0.5 (LLVM 6.0 256 bits), Compiler: GCC 4.8.5 20150623, File-System: xfs, Screen Resolution: 1920x1080
Compiler Notes: --build=x86_64-redhat-linux --disable-libgcj --disable-libunwind-exceptions --enable-__cxa_atexit --enable-bootstrap --enable-checking=release --enable-gnu-indirect-function --enable-gnu-unique-object --enable-initfini-array --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-shared --enable-threads=posix --mandir=/usr/share/man --with-arch_32=x86-64 --with-linker-hash-style=gnu --with-tune=generic
Processor Notes: Scaling Governor: intel_pstate performance
Security Notes: SELinux + KPTI + Load fences __user pointer sanitization + IBRS (kernel) + SSB disabled via prctl and seccomp + PTE Inversion; VMX: SMT vulnerable L1D conditional cache flushes
iPerf
iPerf is a network bandwidth throughput testing software. This test profile is used for automated testing of an iperf client and requires you have access to an iperf server. Learn more via the OpenBenchmarking.org test page.
Nuttcp
Nuttcp is a network performance measurement tool intended for use by network and system managers. Its most basic usage is to determine the raw TCP (or UDP) network layer throughput by transferring memory buffers from a source system across an interconnecting network to a destination system, either transferring data for a specified time interval, or alternatively transferring a specified number of bytes. Learn more via the OpenBenchmarking.org test page.
Ethr
Ethr is a cross-platform Golang-written network performance measurement tool developed by Microsoft that is capable of testing multiple protocols and different measurements. Learn more via the OpenBenchmarking.org test page.
ConnectX: Windows 2019
Processor: 2 x Intel Xeon Gold 6138 @ 2.00GHz (40 Cores / 80 Threads), Motherboard: TYAN S7106, Memory: 12 x 8192 MB 2666MHz Micron 9ASF1G72PZ-2G6B1, Disk: 233GB Samsung SSD 970 EVO 250GB, Graphics: Microsoft Basic Display, Network: Intel I210 Gigabit Connection + Intel I210 Gigabit Connection #2
OS: Microsoft Windows Server 2019 Essentials Build 17763, Kernel: 10.0 (x86_64), Display Driver: 10.0.17763.1, File-System: NTFS, Screen Resolution: 1024x768
Testing initiated at 21 January 2019 11:49 by user Administrator.
ConnectX: FreeBSD 12
Processor: Intel Xeon Gold 6138 @ 2.00GHz (80 Cores), Motherboard: TYAN B7106G24EV4HR, Chipset: Intel Sky Lake-E DMI3 Registers, Memory: 97024MB, Disk: Generic NVMe Device, Graphics: ASPEED Family
OS: FreeBSD, Kernel: 12.0-RELEASE (x86_64), Compiler: Clang 6.0.1 (SVN 335540), File-System: zfs, Screen Resolution: 800x600
Security Notes: KPTI
Testing initiated at 22 January 2019 06:51 by user .
ConnectX: Ubuntu 18.04
Processor: 2 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads), Motherboard: TYAN S7106 (V1.02.B20 BIOS), Chipset: Intel Sky Lake-E DMI3 Registers, Memory: 96256MB, Disk: Samsung SSD 970 EVO 250GB, Graphics: ASPEED Family, Monitor: VE228, Network: 2 x Intel I210 + 2 x QLogic cLOM8214 1/10GbE + Mellanox MT26448
OS: Ubuntu 18.04, Kernel: 4.15.0-43-generic (x86_64), Desktop: GNOME Shell 3.28.3, Display Server: X Server 1.19.6, Display Driver: modesetting 1.19.6, OpenGL: 3.3 Mesa 18.0.5 (LLVM 6.0 256 bits), Compiler: GCC 7.3.0, File-System: ext4, Screen Resolution: 1920x1080
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: intel_pstate powersave
Security Notes: KPTI + __user pointer sanitization + Full generic retpoline IBPB IBRS_FW + SSB disabled via prctl and seccomp + PTE Inversion; VMX: conditional cache flushes SMT vulnerable
Testing initiated at 22 January 2019 12:58 by user phoronix.
ConnectX: Debian 9.6
Processor: 2 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads), Motherboard: TYAN S7106 (V1.02.B20 BIOS), Chipset: Intel Device 2020, Memory: 12 x 8192 MB DDR4-2666MHz Micron 9ASF1G72PZ-2G6B1, Disk: Samsung SSD 970 EVO 250GB, Graphics: LLVMpipe, Network: 2 x Intel I210 + 2 x QLogic cLOM8214 1/10GbE + Mellanox MT26448
OS: Debian 9.6, Kernel: 4.9.0-8-amd64 (x86_64), Desktop: GNOME Shell 3.22.3, Display Server: X Server 1.19.2, OpenGL: 3.3 Mesa 13.0.6 Gallium 0.4 (LLVM 3.9 256 bits), Compiler: GCC 6.3.0 20170516, File-System: ext4, Screen Resolution: 1024x768
Compiler Notes: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v
Processor Notes: Scaling Governor: intel_pstate powersave
Security Notes: KPTI + __user pointer sanitization + Full generic retpoline IBPB IBRS_FW + SSB disabled via prctl and seccomp + PTE Inversion; VMX: conditional cache flushes SMT vulnerable
Testing initiated at 23 January 2019 04:44 by user root.
ConnectX: Ubuntu 18.10
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: intel_pstate powersave
Security Notes: KPTI + __user pointer sanitization + Full generic retpoline IBPB IBRS_FW + SSB disabled via prctl and seccomp + PTE Inversion; VMX: conditional cache flushes SMT vulnerable
Testing initiated at 23 January 2019 15:58 by user phoronix.
ConnectX: Ubuntu 18.10 Tuned
Processor: 2 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads), Motherboard: TYAN S7106 (V1.02.B20 BIOS), Chipset: Intel Sky Lake-E DMI3 Registers, Memory: 96256MB, Disk: Samsung SSD 970 EVO 250GB, Graphics: llvmpipe 93GB, Monitor: VE228, Network: 2 x Intel I210 + 2 x QLogic cLOM8214 1/10GbE + Mellanox MT26448
OS: Ubuntu 18.10, Kernel: 4.18.0-13-generic (x86_64), Desktop: GNOME Shell 3.30.1, Display Server: X Server 1.20.1, Display Driver: modesetting 1.20.1, OpenGL: 3.3 Mesa 18.2.2 (LLVM 7.0 256 bits), Compiler: GCC 8.2.0, File-System: ext4, Screen Resolution: 1920x1080
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: intel_pstate performance
Security Notes: KPTI + __user pointer sanitization + Full generic retpoline IBPB IBRS_FW + SSB disabled via prctl and seccomp + PTE Inversion; VMX: conditional cache flushes SMT vulnerable
Testing initiated at 23 January 2019 17:19 by user phoronix.
ConnectX: Scientific 7
Compiler Notes: --build=x86_64-redhat-linux --disable-libgcj --disable-libunwind-exceptions --enable-__cxa_atexit --enable-bootstrap --enable-checking=release --enable-gnu-indirect-function --enable-gnu-unique-object --enable-initfini-array --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-shared --enable-threads=posix --mandir=/usr/share/man --with-arch_32=x86-64 --with-linker-hash-style=gnu --with-tune=generic
Processor Notes: Scaling Governor: intel_pstate powersave
Security Notes: SELinux + KPTI + Load fences __user pointer sanitization + IBRS (kernel) + SSB disabled via prctl and seccomp + PTE Inversion; VMX: SMT vulnerable L1D conditional cache flushes
Testing initiated at 24 January 2019 18:29 by user phoronix.
ConnectX: Scientific 7 Tuned
Processor: 2 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads), Motherboard: TYAN S7106 (V1.02.B20 BIOS), Chipset: Intel Sky Lake-E DMI3 Registers, Memory: 12 x 8192 MB DDR4-2666MT/s Micron 9ASF1G72PZ-2G6B1, Disk: Samsung SSD 970 EVO 250GB, Graphics: llvmpipe 93GB, Monitor: VE228, Network: 2 x Intel I210 + 2 x QLogic cLOM8214 1/10GbE + Mellanox MT26448
OS: Scientific 7.6, Kernel: 3.10.0-957.1.3.el7.x86_64 (x86_64), Desktop: GNOME Shell 3.28.3, Display Server: X Server 1.20.1, Display Driver: modesetting 1.20.1, OpenGL: 2.1 Mesa 18.0.5 (LLVM 6.0 256 bits), Compiler: GCC 4.8.5 20150623, File-System: xfs, Screen Resolution: 1920x1080
Compiler Notes: --build=x86_64-redhat-linux --disable-libgcj --disable-libunwind-exceptions --enable-__cxa_atexit --enable-bootstrap --enable-checking=release --enable-gnu-indirect-function --enable-gnu-unique-object --enable-initfini-array --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-shared --enable-threads=posix --mandir=/usr/share/man --with-arch_32=x86-64 --with-linker-hash-style=gnu --with-tune=generic
Processor Notes: Scaling Governor: intel_pstate performance
Security Notes: SELinux + KPTI + Load fences __user pointer sanitization + IBRS (kernel) + SSB disabled via prctl and seccomp + PTE Inversion; VMX: SMT vulnerable L1D conditional cache flushes
Testing initiated at 25 January 2019 05:55 by user phoronix.