llamafile xeon

Intel Xeon E E-2488 testing with a Supermicro Super Server X13SCL-F v0123456789 (1.1 BIOS) and ASPEED on Ubuntu 24.04 via the Phoronix Test Suite.

Compare your own system(s) to this result file with the Phoronix Test Suite by running the command: phoronix-test-suite benchmark 2404035-NE-LLAMAFILE41
Jump To Table - Results

View

Do Not Show Noisy Results
Do Not Show Results With Incomplete Data
Do Not Show Results With Little Change/Spread
List Notable Results

Statistics

Show Overall Harmonic Mean(s)
Show Overall Geometric Mean
Show Wins / Losses Counts (Pie Chart)
Normalize Results
Remove Outliers Before Calculating Averages

Graph Settings

Force Line Graphs Where Applicable
Convert To Scalar Where Applicable
Prefer Vertical Bar Graphs

Multi-Way Comparison

Condense Multi-Option Tests Into Single Result Graphs

Table

Show Detailed System Result Table

Run Management

Highlight
Result
Hide
Result
Result
Identifier
Performance Per
Dollar
Date
Run
  Test
  Duration
a
April 03
  17 Minutes
b
April 03
  17 Minutes
c
April 03
  17 Minutes
d
April 03
  18 Minutes
Invert Hiding All Results Option
  18 Minutes

Only show results where is faster than
Only show results matching title/arguments (delimit multiple options with a comma):
Do not show results matching title/arguments (delimit multiple options with a comma):

a

Kernel Notes: Transparent Huge Pages: madvise
Processor Notes: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0x122 - Thermald 2.5.6
Security Notes: gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

b

c

d

Processor: Intel Xeon E E-2488 @ 3.20GHz (8 Cores / 16 Threads), Motherboard: Supermicro Super Server X13SCL-F v0123456789 (1.1 BIOS), Chipset: Intel Raptor Lake-S PCH, Memory: 2 x 32GB DRAM-4800MT/s Micron MTC20C2085S1EC48BA1, Disk: 3201GB Micron_7450_MTFDKCC3T2TFS + 960GB Micron_7450_MTFDKBA960TFR, Graphics: ASPEED, Monitor: VE228, Network: 2 x Intel I210

OS: Ubuntu 24.04, Kernel: 6.8.0-11-generic (x86_64), Desktop: GNOME Shell 45.3, Display Server: X Server 1.21.1.11, Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080

llamafile xeonOpenBenchmarking.orgPhoronix Test SuiteIntel Xeon E E-2488 @ 3.20GHz (8 Cores / 16 Threads)Supermicro Super Server X13SCL-F v0123456789 (1.1 BIOS)Intel Raptor Lake-S PCH2 x 32GB DRAM-4800MT/s Micron MTC20C2085S1EC48BA13201GB Micron_7450_MTFDKCC3T2TFS + 960GB Micron_7450_MTFDKBA960TFRASPEEDVE2282 x Intel I210Ubuntu 24.046.8.0-11-generic (x86_64)GNOME Shell 45.3X Server 1.21.1.11GCC 13.2.0ext41920x1080ProcessorMotherboardChipsetMemoryDiskGraphicsMonitorNetworkOSKernelDesktopDisplay ServerCompilerFile-SystemScreen ResolutionLlamafile Xeon BenchmarksSystem Logs- Transparent Huge Pages: madvise- Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0x122 - Thermald 2.5.6 - gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

abcdResult OverviewPhoronix Test Suite100%100%100%101%101%LlamafileLlamafileLlamafilellava-v1.5-7b-q4 - CPUmistral-7b-instruct-v0.2.Q8_0 - CPUwizardcoder-python-34b-v1.0.Q6_K - CPU

Llamafile

OpenBenchmarking.orgTokens Per Second, More Is BetterLlamafile 0.7Test: llava-v1.5-7b-q4 - Acceleration: CPUabcd3691215SE +/- 0.02, N = 3SE +/- 0.04, N = 3SE +/- 0.08, N = 3SE +/- 0.03, N = 313.2813.1813.2413.22
OpenBenchmarking.orgTokens Per Second, More Is BetterLlamafile 0.7Test: llava-v1.5-7b-q4 - Acceleration: CPUabcd48121620Min: 13.25 / Avg: 13.28 / Max: 13.32Min: 13.12 / Avg: 13.18 / Max: 13.26Min: 13.13 / Avg: 13.24 / Max: 13.39Min: 13.17 / Avg: 13.22 / Max: 13.26

OpenBenchmarking.orgTokens Per Second, More Is BetterLlamafile 0.7Test: mistral-7b-instruct-v0.2.Q8_0 - Acceleration: CPUabcd246810SE +/- 0.01, N = 3SE +/- 0.02, N = 3SE +/- 0.01, N = 3SE +/- 0.01, N = 38.368.338.338.32
OpenBenchmarking.orgTokens Per Second, More Is BetterLlamafile 0.7Test: mistral-7b-instruct-v0.2.Q8_0 - Acceleration: CPUabcd3691215Min: 8.33 / Avg: 8.36 / Max: 8.37Min: 8.3 / Avg: 8.33 / Max: 8.35Min: 8.31 / Avg: 8.33 / Max: 8.35Min: 8.31 / Avg: 8.32 / Max: 8.34

OpenBenchmarking.orgTokens Per Second, More Is BetterLlamafile 0.7Test: wizardcoder-python-34b-v1.0.Q6_K - Acceleration: CPUabcd0.48830.97661.46491.95322.4415SE +/- 0.00, N = 3SE +/- 0.00, N = 3SE +/- 0.00, N = 3SE +/- 0.00, N = 32.172.172.172.17
OpenBenchmarking.orgTokens Per Second, More Is BetterLlamafile 0.7Test: wizardcoder-python-34b-v1.0.Q6_K - Acceleration: CPUabcd246810Min: 2.16 / Avg: 2.17 / Max: 2.17Min: 2.17 / Avg: 2.17 / Max: 2.17Min: 2.17 / Avg: 2.17 / Max: 2.17Min: 2.16 / Avg: 2.17 / Max: 2.17

a

Kernel Notes: Transparent Huge Pages: madvise
Processor Notes: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0x122 - Thermald 2.5.6
Security Notes: gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 3 April 2024 21:05 by user phoronix.

b

Kernel Notes: Transparent Huge Pages: madvise
Processor Notes: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0x122 - Thermald 2.5.6
Security Notes: gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 3 April 2024 21:49 by user phoronix.

c

Kernel Notes: Transparent Huge Pages: madvise
Processor Notes: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0x122 - Thermald 2.5.6
Security Notes: gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 3 April 2024 23:36 by user phoronix.

d

Processor: Intel Xeon E E-2488 @ 3.20GHz (8 Cores / 16 Threads), Motherboard: Supermicro Super Server X13SCL-F v0123456789 (1.1 BIOS), Chipset: Intel Raptor Lake-S PCH, Memory: 2 x 32GB DRAM-4800MT/s Micron MTC20C2085S1EC48BA1, Disk: 3201GB Micron_7450_MTFDKCC3T2TFS + 960GB Micron_7450_MTFDKBA960TFR, Graphics: ASPEED, Monitor: VE228, Network: 2 x Intel I210

OS: Ubuntu 24.04, Kernel: 6.8.0-11-generic (x86_64), Desktop: GNOME Shell 45.3, Display Server: X Server 1.21.1.11, Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080

Kernel Notes: Transparent Huge Pages: madvise
Processor Notes: Scaling Governor: acpi-cpufreq performance (Boost: Enabled) - CPU Microcode: 0x122 - Thermald 2.5.6
Security Notes: gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

Testing initiated at 3 April 2024 23:55 by user phoronix.