Llama.cpp Benchmark - OpenBenchmarking.org

Llama.cpp is a port of Facebook's LLaMA model in C/C++ developed by Georgi Gerganov. Llama.cpp allows the inference of LLaMA and other supported models in C/C++. For CPU inference Llama.cpp supports AVX2/AVX-512, ARM NEON, and other modern ISAs along with features like OpenBLAS usage.

To run this test with the Phoronix Test Suite, the basic command is: phoronix-test-suite benchmark llama-cpp.

* Uploading of benchmark result data to OpenBenchmarking.org is always optional (opt-in) via the Phoronix Test Suite for users wishing to share their results publicly.
** Data based on those opting to upload their test results to OpenBenchmarking.org and users enabling the opt-in anonymous statistics reporting while running benchmarks from an Internet-connected platform.
Data updated weekly as of 22 June 2024.

Revision History

pts/llama-cpp-1.1.0 [View Source] Sun, 02 Jun 2024 10:36:25 GMT
Update against Llama.cpp upstream, switch to Llama 3 model.

pts/llama-cpp-1.0.0 [View Source] Wed, 10 Jan 2024 18:02:20 GMT
Initial commit of llama.cpp CPU benchmark.

Performance Metrics

Llama.cpp b3067

Model: Meta-Llama-3-8B-Instruct-Q8_0.gguf

OpenBenchmarking.org metrics for this test profile configuration based on 50 public results since 2 June 2024 with the latest data as of 19 June 2024.

Below is an overview of the generalized performance for components where there is sufficient statistically significant data based upon user-uploaded results. It is important to keep in mind particularly in the Linux/open-source space there can be vastly different OS configurations, with this overview intended to offer just general guidance as to the performance expectations.

Component

Percentile Rank

# Compatible Public Results

Tokens Per Second (Average)

AMD EPYC 9754 128-Core

97th

17.71 ^{+/- 0.04}

AMD Ryzen Threadripper 7980X 64-Cores

91st

13.10 ^{+/- 0.02}

AMD Ryzen Threadripper 3970X 32-Core

81st

10.34 ^{+/- 0.01}

Mid-Tier

75th

< 10.33

Intel Core i9-14900K

75th

9.49 ^{+/- 0.01}

Intel Core i9-10980XE

65th

8.75 ^{+/- 0.73}

AMD Ryzen 9 7950X 16-Core

57th

8.10 ^{+/- 0.10}

Median

50th

8.05

AMD Ryzen 7 7840U

33rd

7.11 ^{+/- 0.01}

Low-Tier

25th

< 6.24

Intel Core Ultra 7 155H

25th

6.22 ^{+/- 0.02}

Intel Core i7-1185G7

17th

5.76 ^{+/- 0.01}

AMD EPYC 7F32 8-Core

13th

5.48 ^{+/- 0.57}

Based on OpenBenchmarking.org data, the selected test / test configuration (Llama.cpp b3067 - Model: Meta-Llama-3-8B-Instruct-Q8_0.gguf) has an average run-time of 4 minutes. By default this test profile is set to run at least 3 times but may increase if the standard deviation exceeds pre-defined defaults or other calculations deem additional runs necessary for greater statistical accuracy of the result.

Notable Instruction Set Usage

Notable instruction set extensions supported by this test, based on an automatic analysis by the Phoronix Test Suite / OpenBenchmarking.org analytics engine.

Instruction Set

Support

Instructions Detected

Advanced Vector Extensions (AVX)

Used by default on supported hardware.
Found on Intel processors since Sandy Bridge (2011).
Found on AMD processors since Bulldozer (2011).

VZEROUPPER VEXTRACTF128 VPERMILPS VBROADCASTSS VPERM2F128 VINSERTF128 VBROADCASTSD VMASKMOVPS

Advanced Vector Extensions 2 (AVX2)

Used by default on supported hardware.
Found on Intel processors since Haswell (2013).
Found on AMD processors since Excavator (2016).

VPBROADCASTQ VEXTRACTI128 VPBROADCASTD VPERMQ VGATHERQPS VPBROADCASTW VINSERTI128 VPBROADCASTB VPSLLVD VPERMD VPSRLVD VPERM2I128

Advanced Vector Extensions 512 (AVX512)

Used by default on supported hardware.

(ZMM REGISTER USE)

AVX Vector Neural Network Instructions (AVX-VNNI)

Used by default on supported hardware.

VPDPBUSD

FMA (FMA)

Used by default on supported hardware.
Found on Intel processors since Haswell (2013).
Found on AMD processors since Bulldozer (2011).

VFMADD231PS VFMADD132PS VFNMADD231PS VFMADD213PS VFNMADD231SS VFMADD132SS VFNMADD213SS VFMADD132SD VFMADD231SS VFMADD213SS

The test / benchmark does honor compiler flag changes.

Last automated analysis: 6 June 2024

This test profile binary relies on the shared libraries libopenblas.so.0, libm.so.6, libc.so.6, libgfortran.so.5, libquadmath.so.0.

Tested CPU Architectures

This benchmark has been successfully tested on the below mentioned architectures. The CPU architectures listed is where successful OpenBenchmarking.org result uploads occurred, namely for helping to determine if a given test is compatible with various alternative CPU architectures.

june juny 3 Systems - 91 Benchmark Results	Intel Core i9-10980XE - ASRock X299 Steel Legend - Intel Sky Lake-E DMI3 Registers Ubuntu 22.04 - 6.5.0-26-generic - GNOME Shell 42.2
june june 4 Systems - 71 Benchmark Results	AMD Ryzen Threadripper 3970X 32-Core - ASUS ROG ZENITH II EXTREME - AMD Starship Ubuntu 22.04 - 6.5.0-26-generic - GNOME Shell 42.2
mtl june 4 Systems - 28 Benchmark Results	Intel Core Ultra 7 155H - MTL Swift SFG14-72T Coral_MTH - Intel Device 7e7f Ubuntu 22.04 - 6.5.0-35-generic - GNOME Shell 42.9
dkd 3 Systems - 52 Benchmark Results	AMD Ryzen 7 7840U - Framework Laptop 13 - AMD Device 14e8 Ubuntu 22.04 - 6.5.0-35-generic - GNOME Shell 42.9
ai tigerlake 3 Systems - 4 Benchmark Results	Intel Core i7-1185G7 - Dell XPS 13 9310 0DXP1F - Intel Tiger Lake-LP Ubuntu 23.10 - 6.7.0-060700rc5-generic - GNOME Shell 45.1
dddgdjda 3 Systems - 64 Benchmark Results	AMD EPYC 7F32 8-Core - ASRockRack EPYCD8 - AMD Starship Debian 12 - 6.1.0-11-amd64 - X Server
old epyc ai 3 Systems - 9 Benchmark Results	AMD EPYC 7551 32-Core - GIGABYTE MZ31-AR0-00 v01010101 - AMD 17h Debian 12 - 6.1.0-10-amd64 - GCC 12.2.0
slsls 3 Systems - 9 Benchmark Results	Intel Core i9-14900K - ASUS PRIME Z790-P WIFI - Intel Raptor Lake-S PCH Ubuntu 24.04 - 6.8.0-31-generic - GNOME Shell 46.0
kdkdk 5 Systems - 9 Benchmark Results	AMD Ryzen 9 7950X 16-Core - ASUS ROG STRIX X670E-E GAMING WIFI - AMD Device 14d8 Ubuntu 24.04 - 6.8.0-31-generic - GNOME Shell 46.0
ai ai ai 5 Systems - 9 Benchmark Results	AMD Ryzen Threadripper 7980X 64-Cores - System76 Thelio Major - AMD Device 14a4 Ubuntu 24.04 - 6.8.0-060800-generic - GNOME Shell 46.0
lll 2 Systems - 33 Benchmark Results	AMD EPYC 9754 128-Core - AMD Titanite_4G - AMD Device 14a4 Ubuntu 24.04 - 6.8.0-22-generic - GCC 13.2.0
lll 2 Systems - 33 Benchmark Results	AMD EPYC 9754 128-Core - AMD Titanite_4G - AMD Device 14a4 Ubuntu 24.04 - 6.8.0-22-generic - GCC 13.2.0
AMD Ryzen 9 7900 ECC Memory 1 System - 236 Benchmark Results	AMD Ryzen 9 7900 12-Core - ASRockRack 1U4LW-B650/2L2T B650D4U-2L2T/BCM - AMD Device 14d8 Ubuntu 24.04 - 6.8.0-31-generic - GNOME Shell 46.0
newd 1 System - 7 Benchmark Results	AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.8.7-zen1-1-zen - Xfce 4.18
nau 1 System - 3 Benchmark Results	AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.8.2-zen2-1-zen - Xfce 4.18

Llama.cpp

Project Site

Source Repository

Test Created

Last Updated

Test Maintainer

Test Type

Average Install Time

Average Run Time

Test Dependencies

Accolades

Supported Platforms

Revision History

Suites Using This Test

Machine Learning

HPC - High Performance Computing

Large Language Models

Performance Metrics

Llama.cpp b3067

Model: Meta-Llama-3-8B-Instruct-Q8_0.gguf

Notable Instruction Set Usage

Tested CPU Architectures

Recent Test Results

Compare

june juny

june june

mtl june

dkd

ai tigerlake

dddgdjda

old epyc ai

slsls

kdkdk

ai ai ai

lll

lll

AMD Ryzen 9 7900 ECC Memory

newd

nau

Most Popular Test Results