llamafile thelio
AMD Ryzen Threadripper 7980X 64-Cores testing with a System76 Thelio Major (FA Z5 BIOS) and NAVI31 45GB on Pop 22.04 via the Phoronix Test Suite.


a: 

	Processor: AMD Ryzen Threadripper 7980X 64-Cores @ 7.79GHz (64 Cores / 128 Threads), Motherboard: System76 Thelio Major (FA Z5 BIOS), Chipset: AMD Device 14a4, Memory: 4 x 32 GB DRAM-4800MT/s Micron MTC20F1045S1RC48BA2, Disk: 1000GB CT1000T700SSD5, Graphics: NAVI31 45GB (1760/1124MHz), Audio: AMD Device 14cc, Monitor: ASUS VP28U, Network: Aquantia Device 14c0 + Realtek RTL8125 2.5GbE + Intel Wi-Fi 6 AX210/AX211/AX411

	OS: Pop 22.04, Kernel: 6.7.0-060700-generic (x86_64), Desktop: GNOME Shell 42.5, Display Server: X Server 1.21.1.4, OpenGL: 4.6 Mesa 23.3.2-1pop0~1704238321~22.04~36f1d0e (LLVM 15.0.7 DRM 3.56), Vulkan: 1.3.267, Compiler: GCC 11.4.0, File-System: ext4, Screen Resolution: 3840x2160

b: 

	Processor: AMD Ryzen Threadripper 7980X 64-Cores @ 7.79GHz (64 Cores / 128 Threads), Motherboard: System76 Thelio Major (FA Z5 BIOS), Chipset: AMD Device 14a4, Memory: 4 x 32 GB DRAM-4800MT/s Micron MTC20F1045S1RC48BA2, Disk: 1000GB CT1000T700SSD5, Graphics: NAVI31 45GB (1760/1124MHz), Audio: AMD Device 14cc, Monitor: ASUS VP28U, Network: Aquantia Device 14c0 + Realtek RTL8125 2.5GbE + Intel Wi-Fi 6 AX210/AX211/AX411

	OS: Pop 22.04, Kernel: 6.7.0-060700-generic (x86_64), Desktop: GNOME Shell 42.5, Display Server: X Server 1.21.1.4, OpenGL: 4.6 Mesa 23.3.2-1pop0~1704238321~22.04~36f1d0e (LLVM 15.0.7 DRM 3.56), Vulkan: 1.3.267, Compiler: GCC 11.4.0, File-System: ext4, Screen Resolution: 3840x2160

c: 

	Processor: AMD Ryzen Threadripper 7980X 64-Cores @ 7.79GHz (64 Cores / 128 Threads), Motherboard: System76 Thelio Major (FA Z5 BIOS), Chipset: AMD Device 14a4, Memory: 4 x 32 GB DRAM-4800MT/s Micron MTC20F1045S1RC48BA2, Disk: 1000GB CT1000T700SSD5, Graphics: NAVI31 45GB (1760/1124MHz), Audio: AMD Device 14cc, Monitor: ASUS VP28U, Network: Aquantia Device 14c0 + Realtek RTL8125 2.5GbE + Intel Wi-Fi 6 AX210/AX211/AX411

	OS: Pop 22.04, Kernel: 6.7.0-060700-generic (x86_64), Desktop: GNOME Shell 42.5, Display Server: X Server 1.21.1.4, OpenGL: 4.6 Mesa 23.3.2-1pop0~1704238321~22.04~36f1d0e (LLVM 15.0.7 DRM 3.56), Vulkan: 1.3.267, Compiler: GCC 11.4.0, File-System: ext4, Screen Resolution: 3840x2160

d: 

	Processor: AMD Ryzen Threadripper 7980X 64-Cores @ 7.79GHz (64 Cores / 128 Threads), Motherboard: System76 Thelio Major (FA Z5 BIOS), Chipset: AMD Device 14a4, Memory: 4 x 32 GB DRAM-4800MT/s Micron MTC20F1045S1RC48BA2, Disk: 1000GB CT1000T700SSD5, Graphics: NAVI31 45GB (1760/1124MHz), Audio: AMD Device 14cc, Monitor: ASUS VP28U, Network: Aquantia Device 14c0 + Realtek RTL8125 2.5GbE + Intel Wi-Fi 6 AX210/AX211/AX411

	OS: Pop 22.04, Kernel: 6.7.0-060700-generic (x86_64), Desktop: GNOME Shell 42.5, Display Server: X Server 1.21.1.4, OpenGL: 4.6 Mesa 23.3.2-1pop0~1704238321~22.04~36f1d0e (LLVM 15.0.7 DRM 3.56), Vulkan: 1.3.267, Compiler: GCC 11.4.0, File-System: ext4, Screen Resolution: 3840x2160

e: 

	Processor: AMD Ryzen Threadripper 7980X 64-Cores @ 7.79GHz (64 Cores / 128 Threads), Motherboard: System76 Thelio Major (FA Z5 BIOS), Chipset: AMD Device 14a4, Memory: 4 x 32 GB DRAM-4800MT/s Micron MTC20F1045S1RC48BA2, Disk: 1000GB CT1000T700SSD5, Graphics: NAVI31 45GB (1760/1124MHz), Audio: AMD Device 14cc, Monitor: ASUS VP28U, Network: Aquantia Device 14c0 + Realtek RTL8125 2.5GbE + Intel Wi-Fi 6 AX210/AX211/AX411

	OS: Pop 22.04, Kernel: 6.7.0-060700-generic (x86_64), Desktop: GNOME Shell 42.5, Display Server: X Server 1.21.1.4, OpenGL: 4.6 Mesa 23.3.2-1pop0~1704238321~22.04~36f1d0e (LLVM 15.0.7 DRM 3.56), Vulkan: 1.3.267, Compiler: GCC 11.4.0, File-System: ext4, Screen Resolution: 3840x2160


Llamafile 0.6
Test: llava-v1.5-7b-q4 - Acceleration: CPU
Tokens Per Second > Higher Is Better
a . 21.30 |====================================================================
b . 21.37 |====================================================================
c . 21.41 |====================================================================
d . 21.32 |====================================================================
e . 21.45 |====================================================================


Llamafile 0.6
Test: mistral-7b-instruct-v0.2.Q8_0 - Acceleration: CPU
Tokens Per Second > Higher Is Better
a . 13.07 |====================================================================
b . 13.08 |====================================================================
c . 13.03 |====================================================================
d . 13.04 |====================================================================
e . 13.05 |====================================================================


Llamafile 0.6
Test: wizardcoder-python-34b-v1.0.Q6_K - Acceleration: CPU
Tokens Per Second > Higher Is Better
a . 4.40 |=====================================================================
b . 4.40 |=====================================================================
c . 4.41 |=====================================================================
d . 4.41 |=====================================================================
e . 4.40 |=====================================================================