Llama.cpp

Llama.cpp is a port of Facebook's LLaMA model in C/C++ developed by Georgi Gerganov. Llama.cpp allows the inference of LLaMA and other supported models in C/C++. For CPU inference Llama.cpp supports AVX2/AVX-512, ARM NEON, and other modern ISAs along with features like OpenBLAS usage.

To run this test with the Phoronix Test Suite, the basic command is: phoronix-test-suite benchmark llama-cpp.

Project Site

github.com

Source Repository

github.com

Test Created

10 January 2024

Test Maintainer

Michael Larabel

Test Type

System

Average Install Time

37 Seconds

Average Run Time

7 Minutes, 39 Seconds

Test Dependencies

C/C++ Compiler Toolchain + BLAS (Basic Linear Algebra Sub-Routine)

Supported Platforms

* Uploading of benchmark result data to OpenBenchmarking.org is always optional (opt-in) via the Phoronix Test Suite for users wishing to share their results publicly.
** Data based on those opting to upload their test results to OpenBenchmarking.org and users enabling the opt-in anonymous statistics reporting while running benchmarks from an Internet-connected platform.
Data updated weekly as of 21 April 2024.

Revision History

pts/llama-cpp-1.0.0 [View Source] Wed, 10 Jan 2024 18:02:20 GMT
Initial commit of llama.cpp CPU benchmark.

Suites Using This Test

Machine Learning

HPC - High Performance Computing

Large Language Models

Performance Metrics

Analyze Test Configuration:

Llama.cpp b1808

Model: llama-2-13b.Q4_0.gguf

OpenBenchmarking.org metrics for this test profile configuration based on 204 public results since 10 January 2024 with the latest data as of 24 April 2024.

Below is an overview of the generalized performance for components where there is sufficient statistically significant data based upon user-uploaded results. It is important to keep in mind particularly in the Linux/open-source space there can be vastly different OS configurations, with this overview intended to offer just general guidance as to the performance expectations.

Component

Percentile Rank

# Compatible Public Results

Tokens Per Second (Average)

AMD Ryzen Threadripper PRO 7995WX 96-Cores

100th

20.14 ^{+/- 0.12}

AMD Ryzen Threadripper 7980X 64-Cores

97th

18.39 ^{+/- 0.13}

AMD EPYC 8324P 32-Core

92nd

17.88 ^{+/- 0.35}

AMD EPYC 8534PN 32-Core

90th

17.62 ^{+/- 0.44}

AMD EPYC 8534P 64-Core

87th

16.67 ^{+/- 0.05}

2 x AMD EPYC 9684X 96-Core

87th

16.63 ^{+/- 0.69}

AMD EPYC 7R13 48-Core

84th

15.36 ^{+/- 0.79}

ARMv8 Neoverse-N1 128-Core

83rd

13.98 ^{+/- 0.10}

AMD Ryzen Threadripper PRO 5965WX 24-Cores

81st

11.37 ^{+/- 0.18}

AMD Ryzen Threadripper 3970X 32-Core

80th

10.64 ^{+/- 0.01}

Intel Core i9-10980XE

77th

9.79 ^{+/- 0.13}

AMD Ryzen 9 7900X3D 12-Core

76th

8.97

Mid-Tier

75th

< 8.97

AMD Ryzen 9 7900X 12-Core

73rd

8.92

Intel Core i9-14900K

64th

8.75 ^{+/- 0.05}

AMD Ryzen 9 7950X3D 16-Core

63rd

8.74 ^{+/- 0.03}

AMD Ryzen 9 7950X 16-Core

61st

8.71 ^{+/- 0.12}

Intel Xeon Silver 4216

60th

8.68 ^{+/- 0.06}

AMD Ryzen 5 7600X 6-Core

57th

8.36 ^{+/- 0.82}

AMD Ryzen Threadripper 3990X 64-Core

56th

8.05 ^{+/- 0.09}

Intel Core i5-14500

54th

7.99 ^{+/- 0.07}

AMD Ryzen 7 8700G

51st

7.77 ^{+/- 0.79}

Median

50th

7.73

AMD Ryzen 5 8600G

45th

7.53 ^{+/- 0.56}

AMD Ryzen 7 7800X3D 8-Core

41st

7.45

AMD Ryzen 7 7700X 8-Core

40th

7.44

AMD Ryzen 7 7700 8-Core

40th

7.44

AMD Ryzen 7 7840U

30th

7.18 ^{+/- 0.08}

Intel Core i3-14100

29th

7.13 ^{+/- 0.13}

AMD Ryzen 5 5600G

26th

6.36

Intel Core i5-14600K

26th

6.36 ^{+/- 0.13}

Low-Tier

25th

< 6.36

AMD Ryzen 7 5700G

22nd

6.16

AMD EPYC 7F32 8-Core

19th

5.75 ^{+/- 0.68}

Intel Core Ultra 7 155H

17th

5.41 ^{+/- 0.11}

AMD Ryzen 9 5900HX

15th

4.56 ^{+/- 0.02}

Intel Core i7-1185G7

12th

3.84

AMD EPYC 7601 32-Core

9th

2.46 ^{+/- 0.13}

2 x Intel Xeon Max 9480

5th

1.50 ^{+/- 0.03}

Based on OpenBenchmarking.org data, the selected test / test configuration (Llama.cpp b1808 - Model: llama-2-13b.Q4_0.gguf) has an average run-time of 5 minutes. By default this test profile is set to run at least 3 times but may increase if the standard deviation exceeds pre-defined defaults or other calculations deem additional runs necessary for greater statistical accuracy of the result.

Based on public OpenBenchmarking.org results, the selected test / test configuration has an average standard deviation of 0.3%.

Notable Instruction Set Usage

Notable instruction set extensions supported by this test, based on an automatic analysis by the Phoronix Test Suite / OpenBenchmarking.org analytics engine.

Instruction Set

Support

Instructions Detected

Advanced Vector Extensions (AVX)

Used by default on supported hardware.
Found on Intel processors since Sandy Bridge (2011).
Found on AMD processors since Bulldozer (2011).

VZEROUPPER VBROADCASTSS VPERM2F128 VPERMILPS VEXTRACTF128 VINSERTF128 VBROADCASTSD

Advanced Vector Extensions 2 (AVX2)

Used by default on supported hardware.
Found on Intel processors since Haswell (2013).
Found on AMD processors since Excavator (2016).

VPBROADCASTD VPBROADCASTQ VEXTRACTI128 VPERMQ VGATHERQPS VPBROADCASTW VINSERTI128 VPGATHERQQ VPBROADCASTB VPSLLVD VPERMD VPSRLVD VPERM2I128

FMA (FMA)

Used by default on supported hardware.
Found on Intel processors since Haswell (2013).
Found on AMD processors since Bulldozer (2011).

VFMADD231PS VFMADD213PS VFMADD132PS VFMADD132SS VFNMADD213SS VFMADD132SD VFMADD231SS

Advanced Vector Extensions 512 (AVX512)

Requires passing a supported compiler/build flag (verified with targets: cascadelake, sapphirerapids).

(ZMM REGISTER USE)

The test / benchmark does honor compiler flag changes.

Last automated analysis: 11 January 2024

This test profile binary relies on the shared libraries libopenblas.so.0, libm.so.6, libc.so.6, libgfortran.so.5, libquadmath.so.0.

Tested CPU Architectures

This benchmark has been successfully tested on the below mentioned architectures. The CPU architectures listed is where successful OpenBenchmarking.org result uploads occurred, namely for helping to determine if a given test is compatible with various alternative CPU architectures.

CPU Architecture

Kernel Identifier

Verified On

Intel / AMD x86 64-bit

x86_64

(Many Processors)

ARMv8 64-bit

aarch64

ARMv8 Neoverse-N1 128-Core

AMD Ryzen 9 7900 ECC Memory 1 System - 236 Benchmark Results	AMD Ryzen 9 7900 12-Core - ASRockRack 1U4LW-B650/2L2T B650D4U-2L2T/BCM - AMD Device 14d8 Ubuntu 24.04 - 6.8.0-31-generic - GNOME Shell 46.0
newd 1 System - 7 Benchmark Results	AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.8.7-zen1-1-zen - Xfce 4.18
nau 1 System - 3 Benchmark Results	AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.8.2-zen2-1-zen - Xfce 4.18
24.03.13.Pop.2204.ML.test1 1 System - 341 Benchmark Results	AMD Ryzen 9 7950X 16-Core - ASUS ProArt X670E-CREATOR WIFI - AMD Device 14d8 Pop 22.04 - 6.6.10-76060610-generic - GNOME Shell 42.5
neu 2 Systems - 8 Benchmark Results	AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.7.9-zen1-1-zen - X Server 1.21.1.11
neu 1 System - 1 Benchmark Result	AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.7.9-zen1-1-zen - X Server 1.21.1.11
tgl tgl 3 Systems - 79 Benchmark Results	Intel Core i7-1185G7 - Dell XPS 13 9310 0DXP1F - Intel Tiger Lake-LP Ubuntu 23.10 - 6.7.0-060700rc5-generic - GNOME Shell 45.1
xeon febby 2 Systems - 52 Benchmark Results	2 x INTEL XEON PLATINUM 8592+ - Quanta Cloud QuantaGrid D54Q-2U S6Q-MB-MPS - Intel Device 1bce Ubuntu 23.10 - 6.6.0-060600-generic - GCC 13.2.0
hw-mig 2 Systems - 28 Benchmark Results	AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.7.4-zen1-1-zen - Xfce 4.18
hw-mig 2 Systems - 23 Benchmark Results	AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.7.4-zen1-1-zen - Xfce 4.18
hw-mig 2 Systems - 23 Benchmark Results	AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.7.4-zen1-1-zen - Xfce 4.18
hw-mig 2 Systems - 18 Benchmark Results	AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.7.4-zen1-1-zen - Xfce 4.18
hw-mig 2 Systems - 13 Benchmark Results	AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.7.4-zen1-1-zen - Xfce 4.18
New CPU Benchmarks 5 Systems - 587 Benchmark Results	Intel Core i9-14900K - ASUS PRIME Z790-P WIFI - Intel Raptor Lake-S PCH Ubuntu 23.10 - 6.7.3-060703-generic - GNOME Shell 45.2
hw-mig 1 System - 12 Benchmark Results	AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.7.4-zen1-1-zen - Xfce 4.18

AMD Ryzen 7 8700G Linux Benchmarks 6 Systems - 162 Benchmark Results	AMD Ryzen 5 5600G - ASUS TUF GAMING B550M-PLUS - AMD Renoir Ubuntu 23.10 - 6.7.0-060700-generic - GNOME Shell 45.0
AMD Ryzen 5 8500G Linux Benchmarks 18 Systems - 154 Benchmark Results	Intel Core i9-14900K - ASUS PRIME Z790-P WIFI - Intel Device 7a27 Ubuntu 23.10 - 6.7.0-060700-generic - GNOME Shell 45.0
AMD EPYC 8534P 3 Systems - 83 Benchmark Results	AMD EPYC 8534P 64-Core - AMD Cinnabar - AMD Device 14a4 Ubuntu 23.10 - 6.5.0-5-generic - GNOME Shell
New CPU Benchmarks 5 Systems - 587 Benchmark Results	AMD Ryzen 5 8600G - ASRock B650 Pro RS - AMD Device 14e8 Ubuntu 23.10 - 6.7.3-060703-generic - GNOME Shell 45.2
2024 year 4 Systems - 100 Benchmark Results	AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS Pro WS WRX80E-SAGE SE WIFI - AMD Starship Ubuntu 23.10 - 6.5.0-13-generic - GNOME Shell 45.0
AMD EPYC Zen 4C Power Efficiency 5 Systems - 149 Benchmark Results	AMD EPYC 7601 32-Core - TYAN B8026T70AE24HR - AMD 17h Ubuntu 23.10 - 6.6.9-060609-generic - GNOME Shell 45.0
xeon jan 3 Systems - 69 Benchmark Results	Intel Xeon Silver 4216 - TYAN S7100AG2NR - Intel Sky Lake-E DMI3 Registers Debian 12 - 6.1.0-11-amd64 - X Server
rpyc jan 2 Systems - 262 Benchmark Results	AMD EPYC 7F32 8-Core - ASRockRack EPYCD8 - AMD Starship Debian 12 - 6.1.0-11-amd64 - X Server
hw-mig 2 Systems - 28 Benchmark Results	AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.7.4-zen1-1-zen - Xfce 4.18
AMD Ryzen 5 8600G Linux Benchmarks 16 Systems - 168 Benchmark Results	AMD Ryzen 5 5600G - ASUS TUF GAMING B550M-PLUS - AMD Renoir Ubuntu 23.10 - 6.7.0-060700-generic - GNOME Shell 45.0
xeon febby 2 Systems - 52 Benchmark Results	2 x INTEL XEON PLATINUM 8592+ - Quanta Cloud QuantaGrid D54Q-2U S6Q-MB-MPS - Intel Device 1bce Ubuntu 23.10 - 6.6.0-060600-generic - GCC 13.2.0
llama-zen-1 3 Systems - 27 Benchmark Results	AMD EPYC 7601 32-Core - TYAN B8026T70AE24HR - AMD 17h Ubuntu 23.10 - 6.6.9-060609-generic - GNOME Shell 45.0
fg 2 Systems - 31 Benchmark Results	Intel Core i9-10980XE - ASRock X299 Steel Legend - Intel Sky Lake-E DMI3 Registers Ubuntu 22.04 - 6.2.0-39-generic - GNOME Shell 42.2
TR Comparison 2 Systems - 103 Benchmark Results	AMD Ryzen Threadripper 7980X 64-Cores - ASUS Pro WS TRX50-SAGE WIFI - AMD Device 14a4 Pop 22.04 - 6.6.6-76060606-generic - GNOME Shell 42.5
hw-mig 2 Systems - 23 Benchmark Results	AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.7.4-zen1-1-zen - Xfce 4.18