new llamafile

Intel Core i9-14900K testing with a ASUS PRIME Z790-P WIFI (1402 BIOS) and ASUS Intel RPL-S 31GB on Ubuntu 23.10 via the Phoronix Test Suite.

HTML result view exported from: https://openbenchmarking.org/result/2404033-PTS-NEWLLAMA41&sor.

new llamafileProcessorMotherboardChipsetMemoryDiskGraphicsAudioMonitorOSKernelDesktopDisplay ServerOpenGLCompilerFile-SystemScreen ResolutionabcdIntel Core i9-14900K @ 5.70GHz (24 Cores / 32 Threads)ASUS PRIME Z790-P WIFI (1402 BIOS)Intel Device 7a272 x 16GB DRAM-6000MT/s Corsair CMK32GX5M2B6000C36Western Digital WD_BLACK SN850X 1000GB + 2000GB PCIe SSDASUS Intel RPL-S 31GB (1650MHz)Realtek ALC897ASUS VP28UUbuntu 23.106.8.0-phx (x86_64)GNOME Shell 45.1X Server 1.21.1.74.6 Mesa 24.0~git2312240600.c05261~oibaf~m (git-c05261a 2023-12-24 mantic-oibaf-ppa)GCC 13.2.0ext43840x2160OpenBenchmarking.orgKernel Details- Transparent Huge Pages: madviseProcessor Details- Scaling Governor: intel_pstate powersave (EPP: performance) - CPU Microcode: 0x122 - Thermald 2.5.4Security Details- gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + reg_file_data_sampling: Mitigation of Clear Register File + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

new llamafilellamafile: llava-v1.5-7b-q4 - CPUllamafile: mistral-7b-instruct-v0.2.Q8_0 - CPUllamafile: wizardcoder-python-34b-v1.0.Q6_K - CPUabcd14.909.652.6614.859.632.6614.649.612.6514.819.632.66OpenBenchmarking.org

Llamafile

Test: llava-v1.5-7b-q4 - Acceleration: CPU

OpenBenchmarking.orgTokens Per Second, More Is BetterLlamafile 0.7Test: llava-v1.5-7b-q4 - Acceleration: CPUabdc48121620SE +/- 0.02, N = 3SE +/- 0.01, N = 3SE +/- 0.09, N = 3SE +/- 0.06, N = 314.9014.8514.8114.64

Llamafile

Test: mistral-7b-instruct-v0.2.Q8_0 - Acceleration: CPU

OpenBenchmarking.orgTokens Per Second, More Is BetterLlamafile 0.7Test: mistral-7b-instruct-v0.2.Q8_0 - Acceleration: CPUadbc3691215SE +/- 0.03, N = 3SE +/- 0.02, N = 3SE +/- 0.01, N = 3SE +/- 0.03, N = 39.659.639.639.61

Llamafile

Test: wizardcoder-python-34b-v1.0.Q6_K - Acceleration: CPU

OpenBenchmarking.orgTokens Per Second, More Is BetterLlamafile 0.7Test: wizardcoder-python-34b-v1.0.Q6_K - Acceleration: CPUdbac0.59851.1971.79552.3942.9925SE +/- 0.00, N = 3SE +/- 0.00, N = 3SE +/- 0.01, N = 3SE +/- 0.00, N = 32.662.662.662.65

Phoronix Test Suite v10.8.5