xeon max llamafile Tests for a future article. 2 x Intel Xeon Max 9480 testing with a Supermicro SYS-221H-TNR X13DEM v1.10 (1.3 BIOS) and ASPEED on Ubuntu 23.10 via the Phoronix Test Suite.
HTML result view exported from: https://openbenchmarking.org/result/2404036-NE-XEONMAXLL72&sor&export=pdf&grs .
xeon max llamafile Processor Motherboard Chipset Memory Disk Graphics Network OS Kernel Desktop Display Server Compiler File-System Screen Resolution a b 2 x Intel Xeon Max 9480 @ 3.50GHz (112 Cores / 224 Threads) Supermicro SYS-221H-TNR X13DEM v1.10 (1.3 BIOS) Intel Device 1bce 512GB 7682GB INTEL SSDPF2KX076TZ ASPEED 2 x Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ubuntu 23.10 6.9.0-060900rc2-generic (x86_64) GNOME Shell 45.0 X Server 1.21.1.7 GCC 13.2.0 ext4 1024x768 OpenBenchmarking.org Kernel Details - Transparent Huge Pages: madvise Processor Details - Scaling Governor: intel_cpufreq performance - CPU Microcode: 0x2c000290 Security Details - gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + reg_file_data_sampling: Not affected + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Enhanced / Automatic IBRS IBPB: conditional RSB filling PBRSB-eIBRS: SW sequence + srbds: Not affected + tsx_async_abort: Not affected
xeon max llamafile llamafile: llava-v1.5-7b-q4 - CPU llamafile: mistral-7b-instruct-v0.2.Q8_0 - CPU llamafile: wizardcoder-python-34b-v1.0.Q6_K - CPU a b 1.51 3.66 0.59 2.32 3.7 OpenBenchmarking.org
Llamafile Test: llava-v1.5-7b-q4 - Acceleration: CPU OpenBenchmarking.org Tokens Per Second, More Is Better Llamafile 0.7 Test: llava-v1.5-7b-q4 - Acceleration: CPU b a 0.522 1.044 1.566 2.088 2.61 2.32 1.51
Llamafile Test: mistral-7b-instruct-v0.2.Q8_0 - Acceleration: CPU OpenBenchmarking.org Tokens Per Second, More Is Better Llamafile 0.7 Test: mistral-7b-instruct-v0.2.Q8_0 - Acceleration: CPU b a 0.8325 1.665 2.4975 3.33 4.1625 3.70 3.66
Llamafile Test: wizardcoder-python-34b-v1.0.Q6_K - Acceleration: CPU OpenBenchmarking.org Tokens Per Second, More Is Better Llamafile 0.7 Test: wizardcoder-python-34b-v1.0.Q6_K - Acceleration: CPU a 0.1328 0.2656 0.3984 0.5312 0.664 0.59
Phoronix Test Suite v10.8.5