OpenVINO GenAI

This is a benchmark of OpenVINO GenAI that makes use of Intel OpenVINO, a toolkit around neural networks / AI. Complementing the pts/openvino test profile, pts/openvino-genai is focused on Generative AI (GenAI) LLM performance.

OpenVINO GenAI 2024.5

Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU

OpenBenchmarking.org metrics for this test profile configuration based on 95 public results since 24 November 2024 with the latest data as of 20 December 2024.

Below is an overview of the generalized performance for components where there is sufficient statistically significant data based upon user-uploaded results. It is important to keep in mind particularly in the Linux/open-source space there can be vastly different OS configurations, with this overview intended to offer just general guidance as to the performance expectations.

Component

Details

Percentile Rank

# Compatible Public Results

tokens/s (Average)

AMD EPYC 9575F 64-Core

Zen 5 [64 Cores / 128 Threads]

100th

AMD EPYC 9655P 96-Core

Zen 5 [96 Cores / 192 Threads]

86th

56 ^{+/- 1}