mnn ncnn xeon 2 x Intel Xeon Platinum 8380 testing with a Intel M50CYP2SB2U (SE5C6200.86B.0022.D08.2103221623 BIOS) and ASPEED on CentOS Stream 9 via the Phoronix Test Suite.
HTML result view exported from: https://openbenchmarking.org/result/2208133-NE-MNNNCNNXE63&grs&sor&rro .
mnn ncnn xeon Processor Motherboard Chipset Memory Disk Graphics Monitor Network OS Kernel Desktop Display Server Compiler File-System Screen Resolution A B C D E 2 x Intel Xeon Platinum 8380 @ 3.40GHz (80 Cores / 160 Threads) Intel M50CYP2SB2U (SE5C6200.86B.0022.D08.2103221623 BIOS) Intel Device 0998 512GB 7682GB INTEL SSDPF2KX076TZ ASPEED VE228 2 x Intel X710 for 10GBASE-T + 2 x Intel E810-C for QSFP CentOS Stream 9 5.14.0-142.el9.x86_64 (x86_64) GNOME Shell 40.10 X Server GCC 11.3.1 20220421 xfs 1920x1080 OpenBenchmarking.org Kernel Details - Transparent Huge Pages: always Compiler Details - --build=x86_64-redhat-linux --disable-libunwind-exceptions --enable-__cxa_atexit --enable-bootstrap --enable-cet --enable-checking=release --enable-gnu-indirect-function --enable-gnu-unique-object --enable-host-bind-now --enable-host-pie --enable-initfini-array --enable-languages=c,c++,fortran,lto --enable-link-serialization=1 --enable-multilib --enable-offload-targets=nvptx-none --enable-plugin --enable-shared --enable-threads=posix --mandir=/usr/share/man --with-arch_32=x86-64 --with-arch_64=x86-64-v2 --with-build-config=bootstrap-lto --with-gcc-major-version-only --with-linker-hash-style=gnu --with-tune=generic --without-cuda-driver --without-isl Processor Details - Scaling Governor: intel_pstate powersave (EPP: balance_performance) - CPU Microcode: 0xd000363 Security Details - SELinux + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Mitigation of Clear buffers; SMT vulnerable + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling + srbds: Not affected + tsx_async_abort: Not affected
mnn ncnn xeon ncnn: CPU - mobilenet ncnn: CPU - googlenet mnn: resnet-v2-50 ncnn: CPU - squeezenet_ssd ncnn: CPU - FastestDet ncnn: CPU-v2-v2 - mobilenet-v2 ncnn: CPU - blazeface ncnn: CPU - yolov4-tiny ncnn: CPU - resnet50 mnn: mobilenetV3 mnn: inception-v3 ncnn: CPU - regnety_400m mnn: mobilenet-v1-1.0 ncnn: CPU - vision_transformer ncnn: CPU - mnasnet ncnn: CPU - shufflenet-v2 ncnn: CPU-v3-v3 - mobilenet-v3 ncnn: CPU - alexnet ncnn: CPU - resnet18 ncnn: CPU - vgg16 ncnn: CPU - efficientnet-b0 mnn: MobileNetV2_224 mnn: SqueezeNetV1.0 mnn: squeezenetv1.1 A B C D E 21.91 23.74 8.782 26.34 15.33 12.99 7.15 27.80 24.75 1.807 20.690 57.14 2.228 151.78 11.92 13.47 12.03 8.89 13.37 30.91 16.89 3.415 4.121 2.317 21.68 22.53 9.081 26.36 14.98 12.81 7.34 27.25 24.49 1.837 20.812 57.76 2.198 152.63 12.05 13.72 12.21 8.51 13.18 29.05 16.67 3.151 4.213 2.395 22.35 23.11 8.766 27.17 15.56 12.79 7.28 27.28 25.14 1.823 20.853 57.92 2.167 152.79 11.91 13.47 12.25 8.53 13.51 29.63 17.75 3.217 4.295 2.425 22.85 23.52 8.886 26.66 14.92 12.81 7.06 27.38 25.27 1.865 20.829 56.90 2.211 155.31 11.74 13.45 12.12 9.12 13.87 31.20 16.54 3.229 4.219 2.447 21.85 22.64 9.199 25.96 14.93 13.31 7.23 26.74 24.34 1.865 21.346 58.68 2.217 155.90 11.79 13.63 12.21 8.62 13.20 28.68 16.50 3.136 4.321 2.520 OpenBenchmarking.org
NCNN Target: CPU - Model: mobilenet OpenBenchmarking.org ms, Fewer Is Better NCNN 20220729 Target: CPU - Model: mobilenet D C A E B 5 10 15 20 25 SE +/- 0.10, N = 3 SE +/- 0.16, N = 3 SE +/- 0.29, N = 3 SE +/- 0.16, N = 3 SE +/- 0.02, N = 3 22.85 22.35 21.91 21.85 21.68 MIN: 21.59 / MAX: 246.4 MIN: 21.7 / MAX: 47.14 MIN: 21.02 / MAX: 61.03 MIN: 21.29 / MAX: 94.26 MIN: 21.29 / MAX: 45.51 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: googlenet OpenBenchmarking.org ms, Fewer Is Better NCNN 20220729 Target: CPU - Model: googlenet A D C E B 6 12 18 24 30 SE +/- 0.56, N = 3 SE +/- 0.45, N = 3 SE +/- 0.48, N = 3 SE +/- 0.29, N = 3 SE +/- 0.21, N = 3 23.74 23.52 23.11 22.64 22.53 MIN: 21.23 / MAX: 403.67 MIN: 22.04 / MAX: 299.28 MIN: 21.75 / MAX: 82.56 MIN: 21.47 / MAX: 240.52 MIN: 21.73 / MAX: 94.83 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
Mobile Neural Network Model: resnet-v2-50 OpenBenchmarking.org ms, Fewer Is Better Mobile Neural Network 2.0 Model: resnet-v2-50 E B D A C 3 6 9 12 15 SE +/- 0.027, N = 3 SE +/- 0.142, N = 3 SE +/- 0.044, N = 15 SE +/- 0.149, N = 3 SE +/- 0.057, N = 14 9.199 9.081 8.886 8.782 8.766 MIN: 8.97 / MAX: 9.91 MIN: 8.4 / MAX: 24.9 MIN: 8.15 / MAX: 22.2 MIN: 8.31 / MAX: 21.48 MIN: 8.09 / MAX: 22.74 1. (CXX) g++ options: -std=c++11 -O3 -fvisibility=hidden -fomit-frame-pointer -fstrict-aliasing -ffunction-sections -fdata-sections -ffast-math -fno-rtti -fno-exceptions -rdynamic -pthread -ldl
NCNN Target: CPU - Model: squeezenet_ssd OpenBenchmarking.org ms, Fewer Is Better NCNN 20220729 Target: CPU - Model: squeezenet_ssd C D B A E 6 12 18 24 30 SE +/- 0.43, N = 3 SE +/- 0.27, N = 3 SE +/- 0.14, N = 3 SE +/- 0.14, N = 3 SE +/- 0.13, N = 3 27.17 26.66 26.36 26.34 25.96 MIN: 25.76 / MAX: 481.76 MIN: 24.96 / MAX: 198.39 MIN: 25.39 / MAX: 197.69 MIN: 24.99 / MAX: 249.3 MIN: 25.3 / MAX: 49.75 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: FastestDet OpenBenchmarking.org ms, Fewer Is Better NCNN 20220729 Target: CPU - Model: FastestDet C A B E D 4 8 12 16 20 SE +/- 0.45, N = 3 SE +/- 0.14, N = 3 SE +/- 0.20, N = 3 SE +/- 0.09, N = 3 SE +/- 0.22, N = 3 15.56 15.33 14.98 14.93 14.92 MIN: 14.29 / MAX: 72.49 MIN: 14.81 / MAX: 39.13 MIN: 14.48 / MAX: 39.53 MIN: 14.47 / MAX: 37.58 MIN: 14.41 / MAX: 20.23 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU-v2-v2 - Model: mobilenet-v2 OpenBenchmarking.org ms, Fewer Is Better NCNN 20220729 Target: CPU-v2-v2 - Model: mobilenet-v2 E A D B C 3 6 9 12 15 SE +/- 0.30, N = 3 SE +/- 0.14, N = 3 SE +/- 0.09, N = 3 SE +/- 0.04, N = 3 SE +/- 0.05, N = 3 13.31 12.99 12.81 12.81 12.79 MIN: 12.14 / MAX: 236.38 MIN: 12.1 / MAX: 146.34 MIN: 12.34 / MAX: 89.44 MIN: 12.31 / MAX: 37.43 MIN: 12.42 / MAX: 18.33 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: blazeface OpenBenchmarking.org ms, Fewer Is Better NCNN 20220729 Target: CPU - Model: blazeface B C E A D 2 4 6 8 10 SE +/- 0.20, N = 3 SE +/- 0.11, N = 3 SE +/- 0.05, N = 3 SE +/- 0.08, N = 3 SE +/- 0.12, N = 3 7.34 7.28 7.23 7.15 7.06 MIN: 6.93 / MAX: 121.89 MIN: 6.88 / MAX: 9.96 MIN: 6.99 / MAX: 10.09 MIN: 6.83 / MAX: 10.24 MIN: 6.68 / MAX: 8.27 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: yolov4-tiny OpenBenchmarking.org ms, Fewer Is Better NCNN 20220729 Target: CPU - Model: yolov4-tiny A D C B E 7 14 21 28 35 SE +/- 0.43, N = 3 SE +/- 0.26, N = 3 SE +/- 0.03, N = 3 SE +/- 0.25, N = 3 SE +/- 0.19, N = 3 27.80 27.38 27.28 27.25 26.74 MIN: 25.29 / MAX: 281.03 MIN: 25.63 / MAX: 358.69 MIN: 26.44 / MAX: 51.63 MIN: 25.94 / MAX: 352.26 MIN: 25.94 / MAX: 113.75 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: resnet50 OpenBenchmarking.org ms, Fewer Is Better NCNN 20220729 Target: CPU - Model: resnet50 D C A B E 6 12 18 24 30 SE +/- 0.11, N = 3 SE +/- 0.19, N = 3 SE +/- 0.66, N = 3 SE +/- 0.30, N = 3 SE +/- 0.11, N = 3 25.27 25.14 24.75 24.49 24.34 MIN: 23.98 / MAX: 190.68 MIN: 23.76 / MAX: 106.76 MIN: 23.11 / MAX: 123.68 MIN: 23.44 / MAX: 111.14 MIN: 23.27 / MAX: 121.62 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
Mobile Neural Network Model: mobilenetV3 OpenBenchmarking.org ms, Fewer Is Better Mobile Neural Network 2.0 Model: mobilenetV3 E D B C A 0.4196 0.8392 1.2588 1.6784 2.098 SE +/- 0.024, N = 3 SE +/- 0.018, N = 15 SE +/- 0.023, N = 3 SE +/- 0.019, N = 14 SE +/- 0.026, N = 3 1.865 1.865 1.837 1.823 1.807 MIN: 1.79 / MAX: 2.12 MIN: 1.77 / MAX: 4.16 MIN: 1.78 / MAX: 1.98 MIN: 1.67 / MAX: 2.18 MIN: 1.72 / MAX: 4.25 1. (CXX) g++ options: -std=c++11 -O3 -fvisibility=hidden -fomit-frame-pointer -fstrict-aliasing -ffunction-sections -fdata-sections -ffast-math -fno-rtti -fno-exceptions -rdynamic -pthread -ldl
Mobile Neural Network Model: inception-v3 OpenBenchmarking.org ms, Fewer Is Better Mobile Neural Network 2.0 Model: inception-v3 E C D B A 5 10 15 20 25 SE +/- 0.10, N = 3 SE +/- 0.12, N = 14 SE +/- 0.09, N = 15 SE +/- 0.18, N = 3 SE +/- 0.44, N = 3 21.35 20.85 20.83 20.81 20.69 MIN: 18.43 / MAX: 39.41 MIN: 19.34 / MAX: 46.76 MIN: 19.44 / MAX: 41.2 MIN: 19.89 / MAX: 32.91 MIN: 18.2 / MAX: 41.46 1. (CXX) g++ options: -std=c++11 -O3 -fvisibility=hidden -fomit-frame-pointer -fstrict-aliasing -ffunction-sections -fdata-sections -ffast-math -fno-rtti -fno-exceptions -rdynamic -pthread -ldl
NCNN Target: CPU - Model: regnety_400m OpenBenchmarking.org ms, Fewer Is Better NCNN 20220729 Target: CPU - Model: regnety_400m E C B A D 13 26 39 52 65 SE +/- 0.57, N = 3 SE +/- 0.98, N = 3 SE +/- 0.73, N = 3 SE +/- 1.60, N = 3 SE +/- 1.22, N = 3 58.68 57.92 57.76 57.14 56.90 MIN: 54.21 / MAX: 808.52 MIN: 55.21 / MAX: 203.4 MIN: 55.37 / MAX: 447.81 MIN: 53.37 / MAX: 430.33 MIN: 53.28 / MAX: 385.57 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
Mobile Neural Network Model: mobilenet-v1-1.0 OpenBenchmarking.org ms, Fewer Is Better Mobile Neural Network 2.0 Model: mobilenet-v1-1.0 A E D B C 0.5013 1.0026 1.5039 2.0052 2.5065 SE +/- 0.021, N = 3 SE +/- 0.021, N = 3 SE +/- 0.013, N = 15 SE +/- 0.007, N = 3 SE +/- 0.028, N = 13 2.228 2.217 2.211 2.198 2.167 MIN: 2.17 / MAX: 2.38 MIN: 2.16 / MAX: 2.33 MIN: 2.07 / MAX: 5.55 MIN: 2.15 / MAX: 2.45 MIN: 1.83 / MAX: 2.37 1. (CXX) g++ options: -std=c++11 -O3 -fvisibility=hidden -fomit-frame-pointer -fstrict-aliasing -ffunction-sections -fdata-sections -ffast-math -fno-rtti -fno-exceptions -rdynamic -pthread -ldl
NCNN Target: CPU - Model: vision_transformer OpenBenchmarking.org ms, Fewer Is Better NCNN 20220729 Target: CPU - Model: vision_transformer E D C B A 30 60 90 120 150 SE +/- 0.33, N = 3 SE +/- 0.39, N = 3 SE +/- 0.64, N = 3 SE +/- 0.18, N = 3 SE +/- 1.55, N = 3 155.90 155.31 152.79 152.63 151.78 MIN: 146.82 / MAX: 812.77 MIN: 145.23 / MAX: 1014.26 MIN: 145.98 / MAX: 797.33 MIN: 147.33 / MAX: 369.79 MIN: 145.5 / MAX: 656.75 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: mnasnet OpenBenchmarking.org ms, Fewer Is Better NCNN 20220729 Target: CPU - Model: mnasnet B A C E D 3 6 9 12 15 SE +/- 0.16, N = 3 SE +/- 0.10, N = 3 SE +/- 0.10, N = 3 SE +/- 0.08, N = 3 SE +/- 0.13, N = 3 12.05 11.92 11.91 11.79 11.74 MIN: 11.5 / MAX: 157.89 MIN: 11.57 / MAX: 36.77 MIN: 11.32 / MAX: 35.73 MIN: 11.19 / MAX: 63.29 MIN: 11.27 / MAX: 17.51 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: shufflenet-v2 OpenBenchmarking.org ms, Fewer Is Better NCNN 20220729 Target: CPU - Model: shufflenet-v2 B E C A D 4 8 12 16 20 SE +/- 0.31, N = 3 SE +/- 0.26, N = 3 SE +/- 0.16, N = 3 SE +/- 0.10, N = 3 SE +/- 0.10, N = 3 13.72 13.63 13.47 13.47 13.45 MIN: 13.1 / MAX: 82.58 MIN: 12.59 / MAX: 105.4 MIN: 12.86 / MAX: 37.29 MIN: 12.89 / MAX: 36.76 MIN: 12.63 / MAX: 79.26 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU-v3-v3 - Model: mobilenet-v3 OpenBenchmarking.org ms, Fewer Is Better NCNN 20220729 Target: CPU-v3-v3 - Model: mobilenet-v3 C E B D A 3 6 9 12 15 SE +/- 0.06, N = 3 SE +/- 0.16, N = 3 SE +/- 0.20, N = 3 SE +/- 0.03, N = 3 SE +/- 0.14, N = 3 12.25 12.21 12.21 12.12 12.03 MIN: 11.87 / MAX: 36.44 MIN: 11.67 / MAX: 35.66 MIN: 11.64 / MAX: 151.41 MIN: 11.84 / MAX: 34.59 MIN: 11.55 / MAX: 35.77 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: alexnet OpenBenchmarking.org ms, Fewer Is Better NCNN 20220729 Target: CPU - Model: alexnet D A E C B 3 6 9 12 15 SE +/- 0.39, N = 3 SE +/- 0.36, N = 3 SE +/- 0.37, N = 3 SE +/- 0.08, N = 3 SE +/- 0.16, N = 3 9.12 8.89 8.62 8.53 8.51 MIN: 8.11 / MAX: 130.97 MIN: 7.96 / MAX: 119.99 MIN: 7.91 / MAX: 231.7 MIN: 8.13 / MAX: 49.08 MIN: 8.08 / MAX: 100.34 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: resnet18 OpenBenchmarking.org ms, Fewer Is Better NCNN 20220729 Target: CPU - Model: resnet18 D C A E B 4 8 12 16 20 SE +/- 0.61, N = 3 SE +/- 0.15, N = 3 SE +/- 0.24, N = 3 SE +/- 0.12, N = 3 SE +/- 0.11, N = 3 13.87 13.51 13.37 13.20 13.18 MIN: 12.86 / MAX: 18.89 MIN: 12.95 / MAX: 158.24 MIN: 12.55 / MAX: 19.07 MIN: 12.71 / MAX: 82.75 MIN: 12.67 / MAX: 65.17 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: vgg16 OpenBenchmarking.org ms, Fewer Is Better NCNN 20220729 Target: CPU - Model: vgg16 D A C B E 7 14 21 28 35 SE +/- 0.60, N = 3 SE +/- 1.59, N = 3 SE +/- 0.23, N = 3 SE +/- 0.07, N = 3 SE +/- 0.04, N = 3 31.20 30.91 29.63 29.05 28.68 MIN: 28.77 / MAX: 277.95 MIN: 26.06 / MAX: 119.07 MIN: 27.23 / MAX: 201.62 MIN: 27.32 / MAX: 149.49 MIN: 26.99 / MAX: 114.83 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
NCNN Target: CPU - Model: efficientnet-b0 OpenBenchmarking.org ms, Fewer Is Better NCNN 20220729 Target: CPU - Model: efficientnet-b0 C A B D E 4 8 12 16 20 SE +/- 0.73, N = 3 SE +/- 0.14, N = 3 SE +/- 0.14, N = 3 SE +/- 0.29, N = 3 SE +/- 0.30, N = 3 17.75 16.89 16.67 16.54 16.50 MIN: 15.75 / MAX: 590.87 MIN: 16.1 / MAX: 73.41 MIN: 15.93 / MAX: 60.85 MIN: 15.61 / MAX: 86.38 MIN: 15.68 / MAX: 40.87 1. (CXX) g++ options: -O3 -rdynamic -lgomp -lpthread
Mobile Neural Network Model: MobileNetV2_224 OpenBenchmarking.org ms, Fewer Is Better Mobile Neural Network 2.0 Model: MobileNetV2_224 A D C B E 0.7684 1.5368 2.3052 3.0736 3.842 SE +/- 0.143, N = 3 SE +/- 0.047, N = 15 SE +/- 0.049, N = 14 SE +/- 0.076, N = 3 SE +/- 0.105, N = 3 3.415 3.229 3.217 3.151 3.136 MIN: 2.58 / MAX: 12.21 MIN: 2.53 / MAX: 9.07 MIN: 2.68 / MAX: 10.02 MIN: 2.57 / MAX: 8.86 MIN: 2.79 / MAX: 8.34 1. (CXX) g++ options: -std=c++11 -O3 -fvisibility=hidden -fomit-frame-pointer -fstrict-aliasing -ffunction-sections -fdata-sections -ffast-math -fno-rtti -fno-exceptions -rdynamic -pthread -ldl
Mobile Neural Network Model: SqueezeNetV1.0 OpenBenchmarking.org ms, Fewer Is Better Mobile Neural Network 2.0 Model: SqueezeNetV1.0 E C D B A 0.9722 1.9444 2.9166 3.8888 4.861 SE +/- 0.116, N = 3 SE +/- 0.058, N = 14 SE +/- 0.039, N = 15 SE +/- 0.097, N = 3 SE +/- 0.164, N = 3 4.321 4.295 4.219 4.213 4.121 MIN: 3.82 / MAX: 9.63 MIN: 3.71 / MAX: 11.95 MIN: 3.69 / MAX: 15.2 MIN: 3.63 / MAX: 8.6 MIN: 3.72 / MAX: 13.26 1. (CXX) g++ options: -std=c++11 -O3 -fvisibility=hidden -fomit-frame-pointer -fstrict-aliasing -ffunction-sections -fdata-sections -ffast-math -fno-rtti -fno-exceptions -rdynamic -pthread -ldl
Mobile Neural Network Model: squeezenetv1.1 OpenBenchmarking.org ms, Fewer Is Better Mobile Neural Network 2.0 Model: squeezenetv1.1 E D C B A 0.567 1.134 1.701 2.268 2.835 SE +/- 0.079, N = 3 SE +/- 0.027, N = 15 SE +/- 0.054, N = 14 SE +/- 0.110, N = 3 SE +/- 0.101, N = 3 2.520 2.447 2.425 2.395 2.317 MIN: 2.34 / MAX: 6.7 MIN: 2.3 / MAX: 3.94 MIN: 2.11 / MAX: 3.4 MIN: 2.17 / MAX: 4.44 MIN: 2.17 / MAX: 3.61 1. (CXX) g++ options: -std=c++11 -O3 -fvisibility=hidden -fomit-frame-pointer -fstrict-aliasing -ffunction-sections -fdata-sections -ffast-math -fno-rtti -fno-exceptions -rdynamic -pthread -ldl
Phoronix Test Suite v10.8.5