new llamafile

Intel Core i9-14900K testing with a ASUS PRIME Z790-P WIFI (1402 BIOS) and ASUS Intel RPL-S 31GB on Ubuntu 23.10 via the Phoronix Test Suite.

Compare your own system(s) to this result file with the Phoronix Test Suite by running the command: phoronix-test-suite benchmark 2404033-PTS-NEWLLAMA41
Jump To Table - Results

View

Do Not Show Noisy Results
Do Not Show Results With Incomplete Data
Do Not Show Results With Little Change/Spread
List Notable Results

Statistics

Show Overall Harmonic Mean(s)
Show Overall Geometric Mean
Show Wins / Losses Counts (Pie Chart)
Normalize Results
Remove Outliers Before Calculating Averages

Graph Settings

Force Line Graphs Where Applicable
Convert To Scalar Where Applicable
Prefer Vertical Bar Graphs

Multi-Way Comparison

Condense Multi-Option Tests Into Single Result Graphs

Table

Show Detailed System Result Table

Run Management

Highlight
Result
Hide
Result
Result
Identifier
Performance Per
Dollar
Date
Run
  Test
  Duration
a
April 03
  14 Minutes
b
April 03
  14 Minutes
c
April 03
  15 Minutes
d
April 03
  14 Minutes
Invert Hiding All Results Option
  14 Minutes

Only show results where is faster than
Only show results matching title/arguments (delimit multiple options with a comma):
Do not show results matching title/arguments (delimit multiple options with a comma):

a

Kernel Notes: Transparent Huge Pages: madvise
Processor Notes: Scaling Governor: intel_pstate powersave (EPP: performance) - CPU Microcode: 0x122 - Thermald 2.5.4
Security Notes: gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + reg_file_data_sampling: Mitigation of Clear Register File + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

b

c

d

Processor: Intel Core i9-14900K @ 5.70GHz (24 Cores / 32 Threads), Motherboard: ASUS PRIME Z790-P WIFI (1402 BIOS), Chipset: Intel Device 7a27, Memory: 2 x 16GB DRAM-6000MT/s Corsair CMK32GX5M2B6000C36, Disk: Western Digital WD_BLACK SN850X 1000GB + 2000GB PCIe SSD, Graphics: ASUS Intel RPL-S 31GB (1650MHz), Audio: Realtek ALC897, Monitor: ASUS VP28U

OS: Ubuntu 23.10, Kernel: 6.8.0-phx (x86_64), Desktop: GNOME Shell 45.1, Display Server: X Server 1.21.1.7, OpenGL: 4.6 Mesa 24.0~git2312240600.c05261~oibaf~m (git-c05261a 2023-12-24 mantic-oibaf-ppa), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 3840x2160

new llamafileOpenBenchmarking.orgPhoronix Test SuiteIntel Core i9-14900K @ 5.70GHz (24 Cores / 32 Threads)ASUS PRIME Z790-P WIFI (1402 BIOS)Intel Device 7a272 x 16GB DRAM-6000MT/s Corsair CMK32GX5M2B6000C36Western Digital WD_BLACK SN850X 1000GB + 2000GB PCIe SSDASUS Intel RPL-S 31GB (1650MHz)Realtek ALC897ASUS VP28UUbuntu 23.106.8.0-phx (x86_64)GNOME Shell 45.1X Server 1.21.1.74.6 Mesa 24.0~git2312240600.c05261~oibaf~m (git-c05261a 2023-12-24 mantic-oibaf-ppa)GCC 13.2.0ext43840x2160ProcessorMotherboardChipsetMemoryDiskGraphicsAudioMonitorOSKernelDesktopDisplay ServerOpenGLCompilerFile-SystemScreen ResolutionNew Llamafile BenchmarksSystem Logs- Transparent Huge Pages: madvise- Scaling Governor: intel_pstate powersave (EPP: performance) - CPU Microcode: 0x122 - Thermald 2.5.4- gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + reg_file_data_sampling: Mitigation of Clear Register File + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

abcdResult OverviewPhoronix Test Suite100%100%101%101%102%LlamafileLlamafileLlamafilellava-v1.5-7b-q4 - CPUmistral-7b-instruct-v0.2.Q8_0 - CPUwizardcoder-python-34b-v1.0.Q6_K - CPU

Llamafile

OpenBenchmarking.orgTokens Per Second, More Is BetterLlamafile 0.7Test: llava-v1.5-7b-q4 - Acceleration: CPUabcd48121620SE +/- 0.02, N = 3SE +/- 0.01, N = 3SE +/- 0.06, N = 3SE +/- 0.09, N = 314.9014.8514.6414.81
OpenBenchmarking.orgTokens Per Second, More Is BetterLlamafile 0.7Test: llava-v1.5-7b-q4 - Acceleration: CPUabcd48121620Min: 14.87 / Avg: 14.9 / Max: 14.93Min: 14.83 / Avg: 14.85 / Max: 14.87Min: 14.56 / Avg: 14.64 / Max: 14.75Min: 14.63 / Avg: 14.81 / Max: 14.9

OpenBenchmarking.orgTokens Per Second, More Is BetterLlamafile 0.7Test: mistral-7b-instruct-v0.2.Q8_0 - Acceleration: CPUabcd3691215SE +/- 0.03, N = 3SE +/- 0.01, N = 3SE +/- 0.03, N = 3SE +/- 0.02, N = 39.659.639.619.63
OpenBenchmarking.orgTokens Per Second, More Is BetterLlamafile 0.7Test: mistral-7b-instruct-v0.2.Q8_0 - Acceleration: CPUabcd3691215Min: 9.62 / Avg: 9.65 / Max: 9.71Min: 9.61 / Avg: 9.63 / Max: 9.64Min: 9.54 / Avg: 9.61 / Max: 9.65Min: 9.61 / Avg: 9.63 / Max: 9.66

OpenBenchmarking.orgTokens Per Second, More Is BetterLlamafile 0.7Test: wizardcoder-python-34b-v1.0.Q6_K - Acceleration: CPUabcd0.59851.1971.79552.3942.9925SE +/- 0.01, N = 3SE +/- 0.00, N = 3SE +/- 0.00, N = 3SE +/- 0.00, N = 32.662.662.652.66
OpenBenchmarking.orgTokens Per Second, More Is BetterLlamafile 0.7Test: wizardcoder-python-34b-v1.0.Q6_K - Acceleration: CPUabcd246810Min: 2.65 / Avg: 2.66 / Max: 2.67Min: 2.65 / Avg: 2.66 / Max: 2.66Min: 2.65 / Avg: 2.65 / Max: 2.66Min: 2.65 / Avg: 2.66 / Max: 2.66

a

Kernel Notes: Transparent Huge Pages: madvise
Processor Notes: Scaling Governor: intel_pstate powersave (EPP: performance) - CPU Microcode: 0x122 - Thermald 2.5.4
Security Notes: gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + reg_file_data_sampling: Mitigation of Clear Register File + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 3 April 2024 19:36 by user pts.

b

Kernel Notes: Transparent Huge Pages: madvise
Processor Notes: Scaling Governor: intel_pstate powersave (EPP: performance) - CPU Microcode: 0x122 - Thermald 2.5.4
Security Notes: gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + reg_file_data_sampling: Mitigation of Clear Register File + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 3 April 2024 19:52 by user pts.

c

Kernel Notes: Transparent Huge Pages: madvise
Processor Notes: Scaling Governor: intel_pstate powersave (EPP: performance) - CPU Microcode: 0x122 - Thermald 2.5.4
Security Notes: gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + reg_file_data_sampling: Mitigation of Clear Register File + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 3 April 2024 20:06 by user pts.

d

Processor: Intel Core i9-14900K @ 5.70GHz (24 Cores / 32 Threads), Motherboard: ASUS PRIME Z790-P WIFI (1402 BIOS), Chipset: Intel Device 7a27, Memory: 2 x 16GB DRAM-6000MT/s Corsair CMK32GX5M2B6000C36, Disk: Western Digital WD_BLACK SN850X 1000GB + 2000GB PCIe SSD, Graphics: ASUS Intel RPL-S 31GB (1650MHz), Audio: Realtek ALC897, Monitor: ASUS VP28U

OS: Ubuntu 23.10, Kernel: 6.8.0-phx (x86_64), Desktop: GNOME Shell 45.1, Display Server: X Server 1.21.1.7, OpenGL: 4.6 Mesa 24.0~git2312240600.c05261~oibaf~m (git-c05261a 2023-12-24 mantic-oibaf-ppa), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 3840x2160

Kernel Notes: Transparent Huge Pages: madvise
Processor Notes: Scaling Governor: intel_pstate powersave (EPP: performance) - CPU Microcode: 0x122 - Thermald 2.5.4
Security Notes: gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + reg_file_data_sampling: Mitigation of Clear Register File + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 3 April 2024 20:22 by user pts.