Llamafile Benchmark - OpenBenchmarking.org

Mozilla's Llamafile allows distributing and running large language models (LLMs) as a single file. Llamafile aims to make open-source LLMs more accessible to developers and users. Llamafile supports a variety of models, CPUs and GPUs, and other options.

To run this test with the Phoronix Test Suite, the basic command is: phoronix-test-suite benchmark llamafile.

* Uploading of benchmark result data to OpenBenchmarking.org is always optional (opt-in) via the Phoronix Test Suite for users wishing to share their results publicly.
** Data based on those opting to upload their test results to OpenBenchmarking.org and users enabling the opt-in anonymous statistics reporting while running benchmarks from an Internet-connected platform.
Data updated weekly as of 21 April 2024.

Revision History

pts/llamafile-1.1.0 [View Source] Wed, 03 Apr 2024 14:32:58 GMT
Update against Llamafile 0.7.

pts/llamafile-1.0.0 [View Source] Fri, 19 Jan 2024 19:08:42 GMT
Initial commit.

Performance Metrics

Llamafile 0.7

Test: llava-v1.5-7b-q4 - Acceleration: CPU

OpenBenchmarking.org metrics for this test profile configuration based on 63 public results since 3 April 2024 with the latest data as of 24 April 2024.

Below is an overview of the generalized performance for components where there is sufficient statistically significant data based upon user-uploaded results. It is important to keep in mind particularly in the Linux/open-source space there can be vastly different OS configurations, with this overview intended to offer just general guidance as to the performance expectations.

Component

Percentile Rank

# Compatible Public Results

Tokens Per Second (Average)

AMD EPYC 8534P 64-Core

97th

20.77 ^{+/- 0.08}

AMD Ryzen Threadripper 7980X 64-Cores

92nd

20.39 ^{+/- 0.03}

AMD Ryzen Threadripper PRO 5965WX 24-Cores

84th

17.19 ^{+/- 0.01}

AMD Ryzen Threadripper 3970X 32-Core

79th

15.73 ^{+/- 0.05}

Mid-Tier

75th

< 14.90

Intel Core i9-14900K

71st

14.80 ^{+/- 0.11}

2 x AMD EPYC 9684X 96-Core

67th

14.43 ^{+/- 0.15}

Intel Core i9-10980XE

60th

13.36 ^{+/- 0.05}

Intel Xeon E E-2488

54th

13.23 ^{+/- 0.04}

Median

50th

12.99

AMD Ryzen 9 7950X 16-Core

49th

12.95 ^{+/- 0.03}

Intel Xeon Silver 4216

41st

12.82 ^{+/- 0.11}

AMD EPYC 7F32 8-Core

27th

9.87 ^{+/- 1.20}

Low-Tier

25th

< 9.10

Intel Core Ultra 7 155H

24th

9.06 ^{+/- 0.07}

AMD Ryzen 7 PRO 5850U

17th

7.26 ^{+/- 0.03}

AMD EPYC 7551 32-Core

11th

3.79 ^{+/- 0.09}

Based on OpenBenchmarking.org data, the selected test / test configuration (Llamafile 0.7 - Test: llava-v1.5-7b-q4 - Acceleration: CPU) has an average run-time of 2 minutes. By default this test profile is set to run at least 3 times but may increase if the standard deviation exceeds pre-defined defaults or other calculations deem additional runs necessary for greater statistical accuracy of the result.

Based on public OpenBenchmarking.org results, the selected test / test configuration has an average standard deviation of 0.2%.

Tested CPU Architectures

This benchmark has been successfully tested on the below mentioned architectures. The CPU architectures listed is where successful OpenBenchmarking.org result uploads occurred, namely for helping to determine if a given test is compatible with various alternative CPU architectures.

2024-04-13-0917 1 System - 6 Benchmark Results	AMD Ryzen 7 7800X3D 8-Core - Gigabyte B650M K - AMD Device 14d8 Fedora Linux 39 - 6.8.5-201.fc39.x86_64 - GNOME Shell 45.5
dlflsa 3 Systems - 37 Benchmark Results	AMD Ryzen 7 PRO 5850U - LENOVO ThinkPad T14s Gen 2a 20XF004WUS - AMD Renoir Fedora Linux 39 - 6.5.8-300.fc39.x86_64 - GNOME Shell 45.0
dlda 2 Systems - 47 Benchmark Results	Intel Core i9-10980XE - ASRock X299 Steel Legend - Intel Sky Lake-E DMI3 Registers Ubuntu 22.04 - 6.5.0-18-generic - GNOME Shell 42.2
lfld 3 Systems - 58 Benchmark Results	AMD Ryzen Threadripper 3970X 32-Core - ASUS ROG ZENITH II EXTREME - AMD Starship Ubuntu 22.04 - 6.5.0-25-generic - GNOME Shell 42.2
ff 4 Systems - 14 Benchmark Results	2 x AMD EPYC 9684X 96-Core - AMD Titanite_4G - AMD Device 14a4 Ubuntu 23.10 - 6.9.0-060900rc1daily20240327-generic - GCC 13.2.0
sls 3 Systems - 14 Benchmark Results	AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS Pro WS WRX80E-SAGE SE WIFI - AMD Starship Ubuntu 23.10 - 6.5.0-26-generic - GNOME Shell 45.0
x265 ffmpeg llama 3 Systems - 14 Benchmark Results	Intel Xeon Silver 4216 - TYAN S7100AG2NR - Intel Sky Lake-E DMI3 Registers Debian 12 - 6.1.0-11-amd64 - X Server
llama x265 ffmpeg 4 Systems - 13 Benchmark Results	AMD EPYC 7F32 8-Core - ASRockRack EPYCD8 - AMD Starship Debian 12 - 6.1.0-11-amd64 - X Server
epyc fr 2 Systems - 14 Benchmark Results	AMD EPYC 7551 32-Core - GIGABYTE MZ31-AR0-00 v01010101 - AMD 17h Debian 12 - 6.1.0-10-amd64 - GCC 12.2.0
meteorlake llamafile 3 Systems - 3 Benchmark Results	Intel Core Ultra 7 155H - MTL Swift SFG14-72T Coral_MTH - Intel Device 7e7f Ubuntu 23.10 - 6.9.0-060900rc1daily20240327-generic - GNOME Shell 45.2
llamafile sienna 4 Systems - 3 Benchmark Results	AMD EPYC 8534P 64-Core - AMD Cinnabar - AMD Device 14a4 Ubuntu 23.10 - 6.8.1-060801-generic - GNOME Shell 45.2
xeon max llamafile 2 Systems - 3 Benchmark Results	2 x Intel Xeon Max 9480 - Supermicro SYS-221H-TNR X13DEM v1.10 - Intel Device 1bce Ubuntu 23.10 - 6.9.0-060900rc2-generic - GNOME Shell 45.0
llamafile xeon 4 Systems - 3 Benchmark Results	Intel Xeon E E-2488 - Supermicro Super Server X13SCL-F v0123456789 - Intel Raptor Lake-S PCH Ubuntu 24.04 - 6.8.0-11-generic - GNOME Shell 45.3
llamafile 0.7 4 Systems - 3 Benchmark Results	AMD Ryzen 9 7950X 16-Core - ASUS ROG STRIX X670E-E GAMING WIFI - AMD Device 14d8 Ubuntu 23.10 - 6.7.0-060700-generic - GNOME Shell 45.2
llamafile threadripper amd 4 Systems - 3 Benchmark Results	AMD Ryzen Threadripper 7980X 64-Cores - System76 Thelio Major - AMD Device 14a4 Ubuntu 24.04 - 6.8.0-11-generic - GNOME Shell 45.3

epyc fr 2 Systems - 14 Benchmark Results	AMD EPYC 7551 32-Core - GIGABYTE MZ31-AR0-00 v01010101 - AMD 17h Debian 12 - 6.1.0-10-amd64 - GCC 12.2.0
ff 4 Systems - 14 Benchmark Results	2 x AMD EPYC 9684X 96-Core - AMD Titanite_4G - AMD Device 14a4 Ubuntu 23.10 - 6.9.0-060900rc1daily20240327-generic - GCC 13.2.0
xeon max llamafile 2 Systems - 3 Benchmark Results	2 x Intel Xeon Max 9480 - Supermicro SYS-221H-TNR X13DEM v1.10 - Intel Device 1bce Ubuntu 23.10 - 6.9.0-060900rc2-generic - GNOME Shell 45.0
llamafile xeon 4 Systems - 3 Benchmark Results	Intel Xeon E E-2488 - Supermicro Super Server X13SCL-F v0123456789 - Intel Raptor Lake-S PCH Ubuntu 24.04 - 6.8.0-11-generic - GNOME Shell 45.3
new llamafile 4 Systems - 3 Benchmark Results	Intel Core i9-14900K - ASUS PRIME Z790-P WIFI - Intel Device 7a27 Ubuntu 23.10 - 6.8.0-phx - GNOME Shell 45.1
llamafile threadripper amd 4 Systems - 3 Benchmark Results	AMD Ryzen Threadripper 7980X 64-Cores - System76 Thelio Major - AMD Device 14a4 Ubuntu 24.04 - 6.8.0-11-generic - GNOME Shell 45.3
llamafile 0.7 4 Systems - 3 Benchmark Results	AMD Ryzen 9 7950X 16-Core - ASUS ROG STRIX X670E-E GAMING WIFI - AMD Device 14d8 Ubuntu 23.10 - 6.7.0-060700-generic - GNOME Shell 45.2
llama x265 ffmpeg 4 Systems - 13 Benchmark Results	AMD EPYC 7F32 8-Core - ASRockRack EPYCD8 - AMD Starship Debian 12 - 6.1.0-11-amd64 - X Server
dlda 2 Systems - 47 Benchmark Results	Intel Core i9-10980XE - ASRock X299 Steel Legend - Intel Sky Lake-E DMI3 Registers Ubuntu 22.04 - 6.5.0-18-generic - GNOME Shell 42.2
x265 ffmpeg llama 3 Systems - 14 Benchmark Results	Intel Xeon Silver 4216 - TYAN S7100AG2NR - Intel Sky Lake-E DMI3 Registers Debian 12 - 6.1.0-11-amd64 - X Server
llamafile sienna 4 Systems - 3 Benchmark Results	AMD EPYC 8534P 64-Core - AMD Cinnabar - AMD Device 14a4 Ubuntu 23.10 - 6.8.1-060801-generic - GNOME Shell 45.2
dlda 2 Systems - 47 Benchmark Results	Intel Core i9-10980XE - ASRock X299 Steel Legend - Intel Sky Lake-E DMI3 Registers Ubuntu 22.04 - 6.5.0-18-generic - GNOME Shell 42.2
sls 3 Systems - 14 Benchmark Results	AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS Pro WS WRX80E-SAGE SE WIFI - AMD Starship Ubuntu 23.10 - 6.5.0-26-generic - GNOME Shell 45.0
dlflsa 3 Systems - 37 Benchmark Results	AMD Ryzen 7 PRO 5850U - LENOVO ThinkPad T14s Gen 2a 20XF004WUS - AMD Renoir Fedora Linux 39 - 6.5.8-300.fc39.x86_64 - GNOME Shell 45.0
meteorlake llamafile 3 Systems - 3 Benchmark Results	Intel Core Ultra 7 155H - MTL Swift SFG14-72T Coral_MTH - Intel Device 7e7f Ubuntu 23.10 - 6.9.0-060900rc1daily20240327-generic - GNOME Shell 45.2

Llamafile

Project Site

Source Repository

Test Created

Last Updated

Test Maintainer

Test Type

Average Install Time

Average Run Time

Supported Platforms

Revision History

Suites Using This Test

Performance Metrics

Llamafile 0.7

Test: llava-v1.5-7b-q4 - Acceleration: CPU

Tested CPU Architectures

Recent Test Results

Most Popular Test Results