OpenVINO GenAI

This is a benchmark of OpenVINO GenAI that makes use of Intel OpenVINO, a toolkit around neural networks / AI. Complementing the pts/openvino test profile, pts/openvino-genai is focused on Generative AI (GenAI) LLM performance.

OpenVINO GenAI 2024.5

Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPU

OpenBenchmarking.org metrics for this test profile configuration based on 64 public results since 24 November 2024 with the latest data as of 22 December 2024.

Below is an overview of the generalized performance for components where there is sufficient statistically significant data based upon user-uploaded results. It is important to keep in mind particularly in the Linux/open-source space there can be vastly different OS configurations, with this overview intended to offer just general guidance as to the performance expectations.

Component
Details
Percentile Rank
# Compatible Public Results
tokens/s (Average)
AMD EPYC 9575F 64-Core
Zen 5 [64 Cores / 128 Threads]
97th
7
77.6 +/- 0.5
AMD EPYC 9655P 96-Core
Zen 5 [96 Cores / 192 Threads]
76th
11
64.9 +/- 2.4
Mid-Tier
75th
< 64.6
ARMv8 Neoverse-V2 72-Core
69th
4
54.3 +/- 0.8
AMD Ryzen Threadripper 7980X 64-Cores
Zen 4 [64 Cores / 128 Threads]
62nd
3
44.3 +/- 0.1
Intel Core Ultra 9 285K
Arrow Lake [24 Cores / 24 Threads]
58th
5
37.7 +/- 0.1
Median
50th
31.4
AMD Ryzen AI 9 365
Zen 5 [10 Cores / 20 Threads]
48th
3
31.1 +/- 0.6
AMD Ryzen 9 9950X 16-Core
Zen 5 [16 Cores / 32 Threads]
45th
6
30.8 +/- 0.1
AMD Ryzen AI 9 HX 370
Zen 5 [12 Cores / 24 Threads]
33rd
5
28.3 +/- 0.2
AMD Ryzen 7 9800X3D 8-Core
Zen 5 [8 Cores / 16 Threads]
26th
4
26.4
Low-Tier
25th
< 26.4
AMD Ryzen 7 7840HS
Zen 4 [8 Cores / 16 Threads]
22nd
4
24.8
Intel Core i7-1280P
Alder Lake [14 Cores / 20 Threads]
14th
3
21.9 +/- 1.2
ARMv8 Neoverse-N1 128-Core
8th
5
19.6 +/- 1.4