xeon max llamafile

2 x Intel Xeon Max 9480 testing with a Supermicro SYS-221H-TNR X13DEM v1.10 (1.3 BIOS) and ASPEED on Ubuntu 23.10 via the Phoronix Test Suite.

HTML result view exported from: https://openbenchmarking.org/result/2404031-NE-XEONMAXLL89.

xeon max llamafileProcessorMotherboardChipsetMemoryDiskGraphicsNetworkOSKernelDesktopDisplay ServerCompilerFile-SystemScreen Resolutionab2 x Intel Xeon Max 9480 @ 3.50GHz (112 Cores / 224 Threads)Supermicro SYS-221H-TNR X13DEM v1.10 (1.3 BIOS)Intel Device 1bce512GB7682GB INTEL SSDPF2KX076TZASPEED2 x Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200GbUbuntu 23.106.9.0-060900rc2-generic (x86_64)GNOME Shell 45.0X Server 1.21.1.7GCC 13.2.0ext41024x768OpenBenchmarking.orgKernel Details- Transparent Huge Pages: madviseProcessor Details- Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c000290Security Details- gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + reg_file_data_sampling: Not affected + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected

xeon max llamafilellamafile: llava-v1.5-7b-q4 - CPUllamafile: mistral-7b-instruct-v0.2.Q8_0 - CPUllamafile: wizardcoder-python-34b-v1.0.Q6_K - CPUab1.513.660.592.323.7OpenBenchmarking.org

Llamafile

Test: llava-v1.5-7b-q4 - Acceleration: CPU

OpenBenchmarking.orgTokens Per Second, More Is BetterLlamafile 0.7Test: llava-v1.5-7b-q4 - Acceleration: CPUab0.5221.0441.5662.0882.611.512.32

Llamafile

Test: mistral-7b-instruct-v0.2.Q8_0 - Acceleration: CPU

OpenBenchmarking.orgTokens Per Second, More Is BetterLlamafile 0.7Test: mistral-7b-instruct-v0.2.Q8_0 - Acceleration: CPUab0.83251.6652.49753.334.16253.663.70

Llamafile

Test: wizardcoder-python-34b-v1.0.Q6_K - Acceleration: CPU

OpenBenchmarking.orgTokens Per Second, More Is BetterLlamafile 0.7Test: wizardcoder-python-34b-v1.0.Q6_K - Acceleration: CPUa0.13280.26560.39840.53120.6640.59

Phoronix Test Suite v10.8.4