Gigabit Linux vs. FreeBSD vs. Windows Network Benchmarks

Tests by Michael Larabel for a future article.

HTML result view exported from: https://openbenchmarking.org/result/1901253-SP-GIGABIT7499&sro&grs.

ProcessorMotherboardMemoryDiskGraphicsNetworkChipsetMonitorOSKernelDisplay DriverFile-SystemScreen ResolutionCompilerDesktopDisplay ServerOpenGLIntel I210 Windows 2019 FreeBSD 12 Ubuntu 18.04 Debian 9.6 Clear Linux Ubuntu 18.10 Scientific 72 x Intel Xeon Gold 6138 @ 2.00GHz (40 Cores / 80 Threads)TYAN S710612 x 8192 MB 2666MHz Micron 9ASF1G72PZ-2G6B1233GB Samsung SSD 970 EVO 250GBMicrosoft Basic DisplayIntel I210 Gigabit Connection + Intel I210 Gigabit Connection #2Microsoft Windows Server 2019 Essentials Build 1776310.0 (x86_64)10.0.17763.1NTFS1024x768Intel Xeon Gold 6138 @ 2.00GHz (80 Cores)TYAN B7106G24EV4HRIntel Sky Lake-E DMI3 Registers97024MBGeneric NVMe DeviceASPEED FamilyFreeBSD12.0-RELEASE (x86_64)Clang 6.0.1 (SVN 335540)zfs800x6002 x Intel Xeon Gold 6138 @ 3.70GHz (40 Cores / 80 Threads)TYAN S7106 (V1.02.B20 BIOS)96256MBSamsung SSD 970 EVO 250GBllvmpipe 93GBVE2282 x Intel I210 + 2 x QLogic cLOM8214 1/10GbE + Mellanox MT26448Ubuntu 18.044.15.0-43-generic (x86_64)GNOME Shell 3.28.3X Server 1.19.6modesetting 1.19.63.3 Mesa 18.0.5 (LLVM 6.0 256 bits)GCC 7.3.0ext41920x1080Intel Device 202012 x 8192 MB DDR4-2666MHz Micron 9ASF1G72PZ-2G6B1LLVMpipeDebian 9.64.9.0-8-amd64 (x86_64)GNOME Shell 3.22.3X Server 1.19.23.3 Mesa 13.0.6 Gallium 0.4 (LLVM 3.9 256 bits)GCC 6.3.0 201705161024x768Intel Sky Lake-E DMI3 Registers96256MBllvmpipe 93GBVE228Clear Linux OS 274004.20.3-685.native (x86_64)GNOME Shell 3.30.2X Server 1.20.3modesetting 1.20.33.3 Mesa 19.0.0-devel (LLVM 7.0 256 bits)GCC 8.2.1 20180502 + Clang 7.0.1 + LLVM 7.0.11920x1080Ubuntu 18.104.18.0-13-generic (x86_64)GNOME Shell 3.30.1X Server 1.20.1modesetting 1.20.13.3 Mesa 18.2.2 (LLVM 7.0 256 bits)GCC 8.2.012 x 8192 MB DDR4-2666MT/s Micron 9ASF1G72PZ-2G6B1Scientific 7.63.10.0-957.1.3.el7.x86_64 (x86_64)GNOME Shell 3.28.32.1 Mesa 18.0.5 (LLVM 6.0 256 bits)GCC 4.8.5 20150623xfsOpenBenchmarking.orgSecurity Details- Intel I210: FreeBSD 12: KPTI- Intel I210: Ubuntu 18.04: KPTI + __user pointer sanitization + Full generic retpoline IBPB IBRS_FW + SSB disabled via prctl and seccomp + PTE Inversion; VMX: conditional cache flushes SMT vulnerable- Intel I210: Debian 9.6: KPTI + __user pointer sanitization + Full generic retpoline IBPB IBRS_FW + SSB disabled via prctl and seccomp + PTE Inversion; VMX: conditional cache flushes SMT vulnerable- Intel I210: Clear Linux: KPTI + __user pointer sanitization + Full generic retpoline IBPB: conditional IBRS_FW STIBP: conditional RSB filling + SSB disabled via prctl and seccomp + PTE Inversion; VMX: conditional cache flushes SMT vulnerable- Intel I210: Ubuntu 18.10: KPTI + __user pointer sanitization + Full generic retpoline IBPB IBRS_FW + SSB disabled via prctl and seccomp + PTE Inversion; VMX: conditional cache flushes SMT vulnerable- Intel I210: Scientific 7: SELinux + KPTI + Load fences __user pointer sanitization + IBRS (kernel) + SSB disabled via prctl and seccomp + PTE Inversion; VMX: SMT vulnerable L1D conditional cache flushesCompiler Details- Intel I210: Ubuntu 18.04: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic --without-cuda-driver -v - Intel I210: Debian 9.6: --build=x86_64-linux-gnu --disable-browser-plugin --disable-vtable-verify --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-gtk-cairo --enable-java-awt=gtk --enable-java-home --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-arch-directory=amd64 --with-default-libstdcxx-abi=new --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic -v - Intel I210: Clear Linux: --build=x86_64-generic-linux --disable-libmpx --disable-libunwind-exceptions --disable-multiarch --disable-vtable-verify --enable-__cxa_atexit --enable-bootstrap --enable-clocale=gnu --enable-gnu-indirect-function --enable-languages=c,c++,fortran,go --enable-ld=default --enable-libstdcxx-pch --enable-lto --enable-multilib --enable-plugin --enable-shared --enable-threads=posix --exec-prefix=/usr --includedir=/usr/include --target=x86_64-generic-linux --with-arch=westmere --with-glibc-version=2.19 --with-gnu-ld --with-isl --with-ppl=yes --with-tune=haswell - Intel I210: Ubuntu 18.10: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --enable-libmpx --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib --with-tune=generic --without-cuda-driver -v - Intel I210: Scientific 7: --build=x86_64-redhat-linux --disable-libgcj --disable-libunwind-exceptions --enable-__cxa_atexit --enable-bootstrap --enable-checking=release --enable-gnu-indirect-function --enable-gnu-unique-object --enable-initfini-array --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-shared --enable-threads=posix --mandir=/usr/share/man --with-arch_32=x86-64 --with-linker-hash-style=gnu --with-tune=genericProcessor Details- Intel I210: Ubuntu 18.04: Scaling Governor: intel_pstate powersave- Intel I210: Debian 9.6: Scaling Governor: intel_pstate powersave- Intel I210: Clear Linux: Scaling Governor: intel_pstate performance- Intel I210: Ubuntu 18.10: Scaling Governor: intel_pstate powersave- Intel I210: Scientific 7: Scaling Governor: intel_pstate powersaveEnvironment Details- Intel I210: Clear Linux: CFFLAGS=-g-O3-feliminate-unused-debug-types-pipe-Wall-Wp-D_FORTIFY_SOURCE=2-fexceptions-fstack-protector--param=ssp-buffer-size=32-Wl--copy-dt-needed-entries-m64-fasynchronous-unwind-tables-Wp-D_REENTRANT-ftree-loop-distribute-patterns-Wl-z-Wl now-Wl-z-Wl relro-malign-data=abi-fno-semantic-interposition-ftree-vectorize-ftree-loop-vectorize-Wl-sort-common-Wl--enable-new-dtags FFLAGS=-g-O3-feliminate-unused-debug-types-pipe-Wall-Wp-D_FORTIFY_SOURCE=2-fexceptions-fstack-protector--param=ssp-buffer-size=32-Wl--copy-dt-needed-entries-m64-fasynchronous-unwind-tables-Wp-D_REENTRANT-ftree-loop-distribute-patterns-Wl-z-Wl relro-malign-data=abi-fno-semantic-interposition-ftree-vectorize-ftree-loop-vectorize-Wl--enable-new-dtags CXXFLAGS=-g-O3-feliminate-unused-debug-types-pipe-Wall-Wp-D_FORTIFY_SOURCE=2-fexceptions-fstack-protector--param=ssp-buffer-size=32-Wformat-Wformat-security-Wl--copy-dt-needed-entries-m64-fasynchronous-unwind-tables-Wp-D_REENTRANT-ftree-loop-distribute-patterns-Wl-z-Wl relro-fno-semantic-interposition-ffat-lto-objects-fno-signed-zeros-fno-trapping-math-fassociative-math-Wl-sort-common-Wl--enable-new-dtags-mtune=skylake-fvisibility-inlines-hidden-Wl--enable-new-dtags MESA_GLSL_CACHE_DISABLE=0 CFLAGS=-g-O3-feliminate-unused-debug-types-pipe-Wall-Wp-D_FORTIFY_SOURCE=2-fexceptions-fstack-protector--param=ssp-buffer-size=32-Wformat-Wformat-security-Wl--copy-dt-needed-entries-m64-fasynchronous-unwind-tables-Wp-D_REENTRANT-ftree-loop-distribute-patterns-Wl-z-Wl relro-fno-semantic-interposition-ffat-lto-objects-fno-signed-zeros-fno-trapping-math-fassociative-math-Wl-sort-common-Wl--enable-new-dtags-mtune=skylake THEANO_FLAGS=floatX=float32 openmp=true gcc.cxxflags="-ftree-vectorize-mavx"

ethr: 192.168.1.83 - TCP - Connections/s - 1ethr: 192.168.1.83 - TCP - Latency - 64ethr: 192.168.1.83 - TCP - Connections/s - 8ethr: 192.168.1.83 - TCP - Bandwidth - 64ethr: 192.168.1.83 - HTTP - Bandwidth - 1ethr: 192.168.1.83 - TCP - Bandwidth - 8ethr: 192.168.1.83 - UDP - Bandwidth - 8iperf: 192.168.1.83 - 5201 - 60 Seconds - TCP - 1nuttcp: TCP Transfer - Default - 192.168.1.83iperf: 192.168.1.83 - 5201 - 60 Seconds - TCP - 20iperf: 192.168.1.83 - 5201 - 60 Seconds - TCP - 5ethr: 192.168.1.83 - TCP - Latency - 8ethr: 192.168.1.83 - TCP - Latency - 1Intel I210 Windows 2019 FreeBSD 12 Ubuntu 18.04 Debian 9.6 Clear Linux Ubuntu 18.10 Scientific 74146134.8435700986326949961949949950949125.36134.42940940943941667397.384416394230594398294294194194288.3885.92944378.955148798130994296194294194294279.3379.691049764.4343647122733899096193393494194166.4870.89708392.044296397129694398694294194194194.6493.444340102.75421731038302942961941941942941101.30101.61OpenBenchmarking.org

Ethr

Server Address: 192.168.1.83 - Protocol: TCP - Test: Connections/s - Threads: 1

OpenBenchmarking.orgConnections/sec, More Is BetterEthr 2019-01-02Server Address: 192.168.1.83 - Protocol: TCP - Test: Connections/s - Threads: 1Clear LinuxDebian 9.6Scientific 7Ubuntu 18.04Ubuntu 18.10Windows 20192K4K6K8K10KSE +/- 17.64, N = 3SE +/- 169.54, N = 3SE +/- 10.00, N = 3SE +/- 27.28, N = 3SE +/- 36.67, N = 3SE +/- 98.75, N = 121049794434340667370834146

Ethr

Server Address: 192.168.1.83 - Protocol: TCP - Test: Latency - Threads: 64

OpenBenchmarking.orgMicroseconds, Fewer Is BetterEthr 2019-01-02Server Address: 192.168.1.83 - Protocol: TCP - Test: Latency - Threads: 64Clear LinuxDebian 9.6Scientific 7Ubuntu 18.04Ubuntu 18.10Windows 2019306090120150SE +/- 0.36, N = 3SE +/- 3.58, N = 9SE +/- 1.98, N = 3SE +/- 0.03, N = 3SE +/- 1.57, N = 12SE +/- 2.60, N = 1264.4378.95102.7597.3892.04134.84MIN: 57.5 / MAX: 246.61MIN: 60.41 / MAX: 165.1MIN: 85.77 / MAX: 212.99MIN: 86.66 / MAX: 152.16MIN: 73.87 / MAX: 179.33

Ethr

Server Address: 192.168.1.83 - Protocol: TCP - Test: Connections/s - Threads: 8

OpenBenchmarking.orgConnections/sec, More Is BetterEthr 2019-01-02Server Address: 192.168.1.83 - Protocol: TCP - Test: Connections/s - Threads: 8Clear LinuxDebian 9.6Scientific 7Ubuntu 18.04Ubuntu 18.10Windows 201911K22K33K44K55KSE +/- 5451.82, N = 11SE +/- 290.36, N = 3SE +/- 151.69, N = 3SE +/- 35.28, N = 3SE +/- 78.60, N = 3SE +/- 398.04, N = 3436475148742173441634296335700

Ethr

Server Address: 192.168.1.83 - Protocol: TCP - Test: Bandwidth - Threads: 64

OpenBenchmarking.orgMbits/sec, More Is BetterEthr 2019-01-02Server Address: 192.168.1.83 - Protocol: TCP - Test: Bandwidth - Threads: 64Clear LinuxDebian 9.6Scientific 7Ubuntu 18.04Ubuntu 18.10Windows 201930060090012001500SE +/- 86.47, N = 12SE +/- 11.69, N = 12SE +/- 29.25, N = 12SE +/- 0.30, N = 3SE +/- 13.05, N = 6SE +/- 8.22, N = 612279811038942971986

Ethr

Server Address: 192.168.1.83 - Protocol: HTTP - Test: Bandwidth - Threads: 1

OpenBenchmarking.orgMbits/sec, More Is BetterEthr 2019-01-02Server Address: 192.168.1.83 - Protocol: HTTP - Test: Bandwidth - Threads: 1Clear LinuxDebian 9.6Scientific 7Ubuntu 18.04Ubuntu 18.10Windows 201970140210280350SE +/- 1.22, N = 3SE +/- 5.75, N = 12SE +/- 0.37, N = 3SE +/- 2.23, N = 3SE +/- 1.18, N = 3SE +/- 1.18, N = 3338309302305296326

Ethr

Server Address: 192.168.1.83 - Protocol: TCP - Test: Bandwidth - Threads: 8

OpenBenchmarking.orgMbits/sec, More Is BetterEthr 2019-01-02Server Address: 192.168.1.83 - Protocol: TCP - Test: Bandwidth - Threads: 8Clear LinuxDebian 9.6Scientific 7Ubuntu 18.04Ubuntu 18.10Windows 20192004006008001000SE +/- 14.61, N = 12SE +/- 0.26, N = 3SE +/- 0.22, N = 3SE +/- 0.41, N = 3SE +/- 0.56, N = 3SE +/- 0.04, N = 3990942942943943949

Ethr

Server Address: 192.168.1.83 - Protocol: UDP - Test: Bandwidth - Threads: 8

OpenBenchmarking.orgMbits/sec, More Is BetterEthr 2019-01-02Server Address: 192.168.1.83 - Protocol: UDP - Test: Bandwidth - Threads: 8Clear LinuxDebian 9.6Scientific 7Ubuntu 18.04Ubuntu 18.10Windows 20192004006008001000SE +/- 0.04, N = 3SE +/- 0.00, N = 3SE +/- 0.03, N = 3SE +/- 10.35, N = 3SE +/- 4.69, N = 4SE +/- 0.04, N = 3961961961982986961

iPerf

Server Address: 192.168.1.83 - Server Port: 5201 - Duration: 60 Seconds - Test: TCP - Parallel: 1

OpenBenchmarking.orgMbits/sec, More Is BetteriPerf 3.1.3Server Address: 192.168.1.83 - Server Port: 5201 - Duration: 60 Seconds - Test: TCP - Parallel: 1Clear LinuxDebian 9.6FreeBSD 12Scientific 7Ubuntu 18.04Ubuntu 18.10Windows 20192004006008001000933942940941942942949-pipe -fexceptions -fstack-protector -m64 -ffat-lto-objects -fno-signed-zeros -fno-trapping-math -fassociative-math -mtune=skylake1. (CC) gcc options: -O3 -march=native -lm

Nuttcp

Test: TCP Transfer - Default - Server Address: 192.168.1.83

OpenBenchmarking.orgMbits/sec, More Is BetterNuttcp 8.1.4Test: TCP Transfer - Default - Server Address: 192.168.1.83Clear LinuxDebian 9.6FreeBSD 12Scientific 7Ubuntu 18.04Ubuntu 18.10Windows 20192004006008001000SE +/- 0.51, N = 3SE +/- 0.43, N = 3SE +/- 0.02, N = 3SE +/- 0.38, N = 3SE +/- 0.00, N = 3SE +/- 0.42, N = 3SE +/- 0.04, N = 3934941940941941941949gccgccclanggccgccgcc

iPerf

Server Address: 192.168.1.83 - Server Port: 5201 - Duration: 60 Seconds - Test: TCP - Parallel: 20

OpenBenchmarking.orgMbits/sec, More Is BetteriPerf 3.1.3Server Address: 192.168.1.83 - Server Port: 5201 - Duration: 60 Seconds - Test: TCP - Parallel: 20Clear LinuxDebian 9.6FreeBSD 12Scientific 7Ubuntu 18.04Ubuntu 18.10Windows 20192004006008001000SE +/- 0.33, N = 3941942943942941941950-pipe -fexceptions -fstack-protector -m64 -ffat-lto-objects -fno-signed-zeros -fno-trapping-math -fassociative-math -mtune=skylake1. (CC) gcc options: -O3 -march=native -lm

iPerf

Server Address: 192.168.1.83 - Server Port: 5201 - Duration: 60 Seconds - Test: TCP - Parallel: 5

OpenBenchmarking.orgMbits/sec, More Is BetteriPerf 3.1.3Server Address: 192.168.1.83 - Server Port: 5201 - Duration: 60 Seconds - Test: TCP - Parallel: 5Clear LinuxDebian 9.6FreeBSD 12Scientific 7Ubuntu 18.04Ubuntu 18.10Windows 20192004006008001000SE +/- 0.33, N = 3941942941941942941949-pipe -fexceptions -fstack-protector -m64 -ffat-lto-objects -fno-signed-zeros -fno-trapping-math -fassociative-math -mtune=skylake1. (CC) gcc options: -O3 -march=native -lm

Ethr

Server Address: 192.168.1.83 - Protocol: TCP - Test: Latency - Threads: 8

OpenBenchmarking.orgMicroseconds, Fewer Is BetterEthr 2019-01-02Server Address: 192.168.1.83 - Protocol: TCP - Test: Latency - Threads: 8Clear LinuxDebian 9.6Scientific 7Ubuntu 18.04Ubuntu 18.10Windows 2019306090120150SE +/- 3.97, N = 9SE +/- 3.27, N = 12SE +/- 1.78, N = 3SE +/- 2.13, N = 12SE +/- 1.87, N = 3SE +/- 2.43, N = 366.4879.33101.3088.3894.64125.36MIN: 53.82 / MAX: 264.05MIN: 59.76 / MAX: 173.34MIN: 85.89 / MAX: 255.68MIN: 69.53 / MAX: 152.73MIN: 82.43 / MAX: 177.46

Ethr

Server Address: 192.168.1.83 - Protocol: TCP - Test: Latency - Threads: 1

OpenBenchmarking.orgMicroseconds, Fewer Is BetterEthr 2019-01-02Server Address: 192.168.1.83 - Protocol: TCP - Test: Latency - Threads: 1Clear LinuxDebian 9.6Scientific 7Ubuntu 18.04Ubuntu 18.10Windows 2019306090120150SE +/- 4.73, N = 12SE +/- 3.16, N = 12SE +/- 0.79, N = 3SE +/- 2.33, N = 9SE +/- 1.43, N = 12SE +/- 1.93, N = 1270.8979.69101.6185.9293.44134.42MIN: 53.92 / MAX: 271.34MIN: 60.89 / MAX: 173.93MIN: 91.28 / MAX: 202MIN: 68.77 / MAX: 154.8MIN: 76.44 / MAX: 179.97


Phoronix Test Suite v10.8.5