OpenVINO AMX Benchmarks

Benchmarks by Michael Larabel for a future article.

Compare your own system(s) to this result file with the Phoronix Test Suite by running the command: phoronix-test-suite benchmark 2301160-NE-OPENVINOA07
Jump To Table - Results

View

Do Not Show Noisy Results
Do Not Show Results With Incomplete Data
Do Not Show Results With Little Change/Spread
List Notable Results

Statistics

Show Overall Harmonic Mean(s)
Show Overall Geometric Mean
Show Wins / Losses Counts (Pie Chart)
Normalize Results
Remove Outliers Before Calculating Averages

Graph Settings

Force Line Graphs Where Applicable
Convert To Scalar Where Applicable
Prefer Vertical Bar Graphs
No Box Plots
On Line Graphs With Missing Data, Connect The Line Gaps

Multi-Way Comparison

Condense Multi-Option Tests Into Single Result Graphs

Table

Show Detailed System Result Table

Run Management

Highlight
Result
Hide
Result
Result
Identifier
View Logs
Performance Per
Dollar
Date
Run
  Test
  Duration
AVX512_CORE_AMX
January 16 2023
  28 Minutes
AVX512_CORE_FP16
January 16 2023
  26 Minutes
AVX512_CORE_BF16
January 16 2023
  26 Minutes
AVX512_CORE_VNNI
January 16 2023
  26 Minutes
AVX512_CORE
January 16 2023
  26 Minutes
Invert Hiding All Results Option
  27 Minutes

Only show results where is faster than
Only show results matching title/arguments (delimit multiple options with a comma):
Do not show results matching title/arguments (delimit multiple options with a comma):

AVX512_CORE_AMX

Kernel Notes: Transparent Huge Pages: madvise
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: intel_pstate performance (EPP: performance) - CPU Microcode: 0x2b0000c0
Python Notes: Python 3.10.6
Security Notes: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

AVX512_CORE_FP16

AVX512_CORE_BF16

AVX512_CORE_VNNI

AVX512_CORE

Processor: 2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS), Chipset: Intel Device 1bce, Memory: 16 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG, Disk: 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96, Graphics: ASPEED, Monitor: VGA HDMI, Network: 4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-T

OS: Ubuntu 22.04, Kernel: 6.1.4-060104-generic (x86_64), Desktop: GNOME Shell 42.2, Display Server: X Server 1.21.1.3, Vulkan: 1.2.204, Compiler: GCC 11.3.0 + Clang 14.0.0-1ubuntu1, File-System: ext4, Screen Resolution: 1920x1080

OpenVINO AMX BenchmarksOpenBenchmarking.orgPhoronix Test Suite2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads)Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS)Intel Device 1bce16 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96ASPEEDVGA HDMI4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-TUbuntu 22.046.1.4-060104-generic (x86_64)GNOME Shell 42.2X Server 1.21.1.31.2.204GCC 11.3.0 + Clang 14.0.0-1ubuntu1ext41920x1080ProcessorMotherboardChipsetMemoryDiskGraphicsMonitorNetworkOSKernelDesktopDisplay ServerVulkanCompilerFile-SystemScreen ResolutionOpenVINO AMX Benchmarks PerformanceSystem Logs- Transparent Huge Pages: madvise- --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v - Scaling Governor: intel_pstate performance (EPP: performance) - CPU Microcode: 0x2b0000c0 - Python 3.10.6- itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

AVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_COREResult OverviewPhoronix Test Suite100%187%273%360%447%OpenVINOOpenVINOOpenVINOOpenVINOOpenVINOOpenVINOOpenVINOOpenVINOF.D.F.I - CPUW.P.D.F.I - CPUW.P.D.F.I - CPUP.V.B.D.F - CPUM.T.E.T.D.F - CPUM.T.E.T.D.F - CPUF.D.F.I - CPUP.V.B.D.F - CPU

OpenVINO AMX Benchmarksopenvino: Face Detection FP16-INT8 - CPUopenvino: Face Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUopenvino: Weld Porosity Detection FP16-INT8 - CPUopenvino: Person Vehicle Bike Detection FP16 - CPUopenvino: Person Vehicle Bike Detection FP16 - CPUopenvino: Machine Translation EN To DE FP16 - CPUopenvino: Machine Translation EN To DE FP16 - CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE282.25424.1623066.635.184778.3425.09724.1082.79146.08409.818897.1013.462730.0721.95321.28186.53146.06409.848918.6213.432767.1621.66314.61190.49146.02409.958909.0513.441821.0932.93387.62154.5563.21942.795835.3920.541820.2732.94378.72158.26OpenBenchmarking.org

OpenVINO

This is a test of the Intel OpenVINO, a toolkit around neural networks, using its built-in benchmarking support and analyzing the throughput and latency for various models. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE60120180240300SE +/- 2.56, N = 3SE +/- 0.29, N = 3SE +/- 0.22, N = 3SE +/- 0.30, N = 3SE +/- 0.03, N = 3282.25146.08146.06146.0263.211. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl
OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE50100150200250Min: 279.46 / Avg: 282.25 / Max: 287.36Min: 145.5 / Avg: 146.08 / Max: 146.38Min: 145.62 / Avg: 146.06 / Max: 146.32Min: 145.58 / Avg: 146.02 / Max: 146.6Min: 63.16 / Avg: 63.21 / Max: 63.261. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE2004006008001000SE +/- 3.75, N = 3SE +/- 0.81, N = 3SE +/- 0.65, N = 3SE +/- 0.81, N = 3SE +/- 0.30, N = 3424.16409.81409.84409.95942.79MIN: 153.43 / MAX: 1952.95MIN: 203.37 / MAX: 909.92MIN: 208.02 / MAX: 643.02MIN: 216.25 / MAX: 615.53MIN: 383.12 / MAX: 1157.461. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl
MinAvgMaxAVX512_CORE_AMX190025613507AVX512_CORE_FP16123123743508AVX512_CORE_BF16190024013505AVX512_CORE_VNNI112623933512AVX512_CORE190026493515OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxAVX512_CORE_AMX208616761AVX512_CORE_FP16210641748AVX512_CORE_BF16104641758AVX512_CORE_VNNI201640752AVX512_CORE214643759OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. AVX512_CORE_AMX: Approximate power consumption of 49863 Joules per run.2. AVX512_CORE_FP16: Approximate power consumption of 45951 Joules per run.3. AVX512_CORE_BF16: Approximate power consumption of 45477 Joules per run.4. AVX512_CORE_VNNI: Approximate power consumption of 45633 Joules per run.5. AVX512_CORE: Approximate power consumption of 45679 Joules per run.
MinAvgMaxAVX512_CORE_AMX30.048.055.0AVX512_CORE_FP1634.052.357.0AVX512_CORE_BF1631.051.156.0AVX512_CORE_VNNI35.052.659.0AVX512_CORE33.052.157.0OpenBenchmarking.orgCelsius, Fewer Is BetterOpenVINO 2022.3CPU Temperature Monitor1632486480
OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Face Detection FP16-INT8 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE170340510680850Min: 416.68 / Avg: 424.16 / Max: 428.34Min: 408.97 / Avg: 409.81 / Max: 411.42Min: 409.12 / Avg: 409.84 / Max: 411.13Min: 408.4 / Avg: 409.95 / Max: 411.13Min: 942.2 / Avg: 942.79 / Max: 943.091. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE5K10K15K20K25KSE +/- 92.31, N = 3SE +/- 24.48, N = 3SE +/- 85.46, N = 3SE +/- 52.22, N = 3SE +/- 78.19, N = 323066.638897.108918.628909.055835.391. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl
OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE4K8K12K16K20KMin: 22889.05 / Avg: 23066.63 / Max: 23199.14Min: 8866.78 / Avg: 8897.1 / Max: 8945.56Min: 8823.84 / Avg: 8918.62 / Max: 9089.19Min: 8837.74 / Avg: 8909.05 / Max: 9010.78Min: 5687.62 / Avg: 5835.39 / Max: 5953.591. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE510152025SE +/- 0.02, N = 3SE +/- 0.04, N = 3SE +/- 0.13, N = 3SE +/- 0.08, N = 3SE +/- 0.28, N = 35.1813.4613.4313.4420.54MIN: 1.43 / MAX: 102.62MIN: 3.82 / MAX: 88.11MIN: 3.72 / MAX: 102.63MIN: 3.47 / MAX: 112.16MIN: 9.17 / MAX: 1141. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl
MinAvgMaxAVX512_CORE_AMX190028603501AVX512_CORE_FP16190029283512AVX512_CORE_BF16190028913512AVX512_CORE_VNNI190029033511AVX512_CORE190026743503OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxAVX512_CORE_AMX217663757AVX512_CORE_FP16210665756AVX512_CORE_BF16210667769AVX512_CORE_VNNI211665760AVX512_CORE212668759OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. AVX512_CORE_AMX: Approximate power consumption of 43514 Joules per run.2. AVX512_CORE_FP16: Approximate power consumption of 43250 Joules per run.3. AVX512_CORE_BF16: Approximate power consumption of 43325 Joules per run.4. AVX512_CORE_VNNI: Approximate power consumption of 43434 Joules per run.5. AVX512_CORE: Approximate power consumption of 42944 Joules per run.
MinAvgMaxAVX512_CORE_AMX39.052.255.0AVX512_CORE_FP1640.055.258.0AVX512_CORE_BF1638.054.658.0AVX512_CORE_VNNI38.054.858.0AVX512_CORE39.054.958.0OpenBenchmarking.orgCelsius, Fewer Is BetterOpenVINO 2022.3CPU Temperature Monitor1632486480
OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Weld Porosity Detection FP16-INT8 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE510152025Min: 5.15 / Avg: 5.18 / Max: 5.22Min: 13.39 / Avg: 13.46 / Max: 13.51Min: 13.18 / Avg: 13.43 / Max: 13.57Min: 13.29 / Avg: 13.44 / Max: 13.55Min: 20.13 / Avg: 20.54 / Max: 21.071. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Person Vehicle Bike Detection FP16 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE10002000300040005000SE +/- 14.65, N = 3SE +/- 11.17, N = 3SE +/- 16.62, N = 3SE +/- 7.34, N = 3SE +/- 8.67, N = 34778.342730.072767.161821.091820.271. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl
OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Person Vehicle Bike Detection FP16 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE8001600240032004000Min: 4755.83 / Avg: 4778.34 / Max: 4805.85Min: 2711.92 / Avg: 2730.07 / Max: 2750.42Min: 2734.02 / Avg: 2767.16 / Max: 2786.05Min: 1806.7 / Avg: 1821.09 / Max: 1830.81Min: 1807.15 / Avg: 1820.27 / Max: 1836.651. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Person Vehicle Bike Detection FP16 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE816243240SE +/- 0.08, N = 3SE +/- 0.09, N = 3SE +/- 0.13, N = 3SE +/- 0.13, N = 3SE +/- 0.16, N = 325.0921.9521.6632.9332.94MIN: 11.66 / MAX: 132.09MIN: 11.62 / MAX: 120.04MIN: 13.42 / MAX: 101.56MIN: 12.7 / MAX: 103.82MIN: 12.45 / MAX: 111.251. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl
MinAvgMaxAVX512_CORE_AMX190029653501AVX512_CORE_FP16190027393511AVX512_CORE_BF16190027553513AVX512_CORE_VNNI190029343508AVX512_CORE190029103513OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxAVX512_CORE_AMX213640755AVX512_CORE_FP16132653756AVX512_CORE_BF16210652754AVX512_CORE_VNNI210655748AVX512_CORE212655749OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. AVX512_CORE_AMX: Approximate power consumption of 45870 Joules per run.2. AVX512_CORE_FP16: Approximate power consumption of 43951 Joules per run.3. AVX512_CORE_BF16: Approximate power consumption of 44098 Joules per run.4. AVX512_CORE_VNNI: Approximate power consumption of 44322 Joules per run.5. AVX512_CORE: Approximate power consumption of 44311 Joules per run.
MinAvgMaxAVX512_CORE_AMX39.051.955.0AVX512_CORE_FP1639.052.857.0AVX512_CORE_BF1639.052.557.0AVX512_CORE_VNNI39.052.055.0AVX512_CORE40.052.155.0OpenBenchmarking.orgCelsius, Fewer Is BetterOpenVINO 2022.3CPU Temperature Monitor1632486480
OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Person Vehicle Bike Detection FP16 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE714212835Min: 24.95 / Avg: 25.09 / Max: 25.21Min: 21.79 / Avg: 21.95 / Max: 22.1Min: 21.51 / Avg: 21.66 / Max: 21.92Min: 32.75 / Avg: 32.93 / Max: 33.19Min: 32.65 / Avg: 32.94 / Max: 33.181. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Machine Translation EN To DE FP16 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE160320480640800SE +/- 5.21, N = 3SE +/- 0.77, N = 3SE +/- 3.03, N = 3SE +/- 2.49, N = 3SE +/- 5.28, N = 3724.10321.28314.61387.62378.721. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl
OpenBenchmarking.orgFPS, More Is BetterOpenVINO 2022.3Model: Machine Translation EN To DE FP16 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE130260390520650Min: 714.06 / Avg: 724.1 / Max: 731.53Min: 319.99 / Avg: 321.28 / Max: 322.65Min: 308.57 / Avg: 314.61 / Max: 318.03Min: 383.26 / Avg: 387.62 / Max: 391.9Min: 369.12 / Avg: 378.72 / Max: 387.311. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Machine Translation EN To DE FP16 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE4080120160200SE +/- 0.60, N = 3SE +/- 0.45, N = 3SE +/- 1.85, N = 3SE +/- 1.04, N = 3SE +/- 2.22, N = 382.79186.53190.49154.55158.26MIN: 31.64 / MAX: 1588.97MIN: 92.82 / MAX: 808.7MIN: 117.21 / MAX: 1606.21MIN: 90.51 / MAX: 1925.11MIN: 62.25 / MAX: 1819.891. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl
MinAvgMaxAVX512_CORE_AMX190027923516AVX512_CORE_FP16190026563508AVX512_CORE_BF16190027103512AVX512_CORE_VNNI190025683509AVX512_CORE190025903518OpenBenchmarking.orgMegahertz, More Is BetterOpenVINO 2022.3CPU Peak Freq (Highest CPU Core Frequency) Monitor10002000300040005000
MinAvgMaxAVX512_CORE_AMX212644742AVX512_CORE_FP16211652756AVX512_CORE_BF16209650756AVX512_CORE_VNNI208648751AVX512_CORE211648753OpenBenchmarking.orgWatts, Fewer Is BetterOpenVINO 2022.3CPU Power Consumption Monitor20040060080010001. AVX512_CORE_AMX: Approximate power consumption of 43344 Joules per run.2. AVX512_CORE_FP16: Approximate power consumption of 44529 Joules per run.3. AVX512_CORE_BF16: Approximate power consumption of 44210 Joules per run.4. AVX512_CORE_VNNI: Approximate power consumption of 44091 Joules per run.5. AVX512_CORE: Approximate power consumption of 44259 Joules per run.
MinAvgMaxAVX512_CORE_AMX39.051.555.0AVX512_CORE_FP1638.052.656.0AVX512_CORE_BF1639.052.657.0AVX512_CORE_VNNI39.052.456.0AVX512_CORE40.052.456.0OpenBenchmarking.orgCelsius, Fewer Is BetterOpenVINO 2022.3CPU Temperature Monitor1632486480
OpenBenchmarking.orgms, Fewer Is BetterOpenVINO 2022.3Model: Machine Translation EN To DE FP16 - Device: CPUAVX512_CORE_AMXAVX512_CORE_FP16AVX512_CORE_BF16AVX512_CORE_VNNIAVX512_CORE306090120150Min: 81.91 / Avg: 82.79 / Max: 83.94Min: 185.74 / Avg: 186.53 / Max: 187.3Min: 188.39 / Avg: 190.49 / Max: 194.17Min: 152.75 / Avg: 154.55 / Max: 156.36Min: 154.68 / Avg: 158.26 / Max: 162.331. (CXX) g++ options: -fsigned-char -ffunction-sections -fdata-sections -O3 -fno-strict-overflow -fwrapv -pie -ldl

8 Results Shown

OpenVINO:
  Face Detection FP16-INT8 - CPU:
    FPS
    ms
  Weld Porosity Detection FP16-INT8 - CPU:
    FPS
    ms
  Person Vehicle Bike Detection FP16 - CPU:
    FPS
    ms
  Machine Translation EN To DE FP16 - CPU:
    FPS
    ms

AVX512_CORE_AMX

Kernel Notes: Transparent Huge Pages: madvise
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: intel_pstate performance (EPP: performance) - CPU Microcode: 0x2b0000c0
Python Notes: Python 3.10.6
Security Notes: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 16 January 2023 10:56 by user root.

AVX512_CORE_FP16

Kernel Notes: Transparent Huge Pages: madvise
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: intel_pstate performance (EPP: performance) - CPU Microcode: 0x2b0000c0
Python Notes: Python 3.10.6
Security Notes: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 16 January 2023 11:18 by user root.

AVX512_CORE_BF16

Kernel Notes: Transparent Huge Pages: madvise
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: intel_pstate performance (EPP: performance) - CPU Microcode: 0x2b0000c0
Python Notes: Python 3.10.6
Security Notes: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 16 January 2023 11:55 by user root.

AVX512_CORE_VNNI

Kernel Notes: Transparent Huge Pages: madvise
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: intel_pstate performance (EPP: performance) - CPU Microcode: 0x2b0000c0
Python Notes: Python 3.10.6
Security Notes: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 16 January 2023 12:10 by user root.

AVX512_CORE

Processor: 2 x Intel Xeon Platinum 8490H @ 3.50GHz (120 Cores / 240 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3A10.uh BIOS), Chipset: Intel Device 1bce, Memory: 16 x 64 GB 4800MT/s Samsung M321R8GA0BB0-CQKEG, Disk: 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 960GB INTEL SSDSC2KG96, Graphics: ASPEED, Monitor: VGA HDMI, Network: 4 x Intel E810-C for QSFP + 2 x Intel X710 for 10GBASE-T

OS: Ubuntu 22.04, Kernel: 6.1.4-060104-generic (x86_64), Desktop: GNOME Shell 42.2, Display Server: X Server 1.21.1.3, Vulkan: 1.2.204, Compiler: GCC 11.3.0 + Clang 14.0.0-1ubuntu1, File-System: ext4, Screen Resolution: 1920x1080

Kernel Notes: Transparent Huge Pages: madvise
Compiler Notes: --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v
Processor Notes: Scaling Governor: intel_pstate performance (EPP: performance) - CPU Microcode: 0x2b0000c0
Python Notes: Python 3.10.6
Security Notes: itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 16 January 2023 12:25 by user root.