llama AMD Ryzen 7 9800X3D 8-Core testing with a ASRock X870E Taichi (3.12.AS02 BIOS) and AMD Radeon PRO W7500 8GB on Ubuntu 24.04 via the Phoronix Test Suite. a: Processor: AMD Ryzen 7 9800X3D 8-Core @ 6.22GHz (8 Cores / 16 Threads), Motherboard: ASRock X870E Taichi (3.12.AS02 BIOS), Chipset: AMD Device 14d8, Memory: 2 x 16GB DDR5-6000MT/s F5-6000J2836G16G, Disk: Western Digital WD_BLACK SN850X 2000GB, Graphics: AMD Radeon PRO W7500 8GB, Audio: AMD Navi 31 HDMI/DP, Monitor: DELL U2723QE, Network: Realtek Device 8126 + MEDIATEK Device 0717 OS: Ubuntu 24.04, Kernel: 6.8.0-49-generic (x86_64), Desktop: GNOME Shell 46.0, Display Server: X Server 1.21.1.11 + Wayland, OpenGL: 4.6 Mesa 24.2.0-devel (LLVM 18.1.7 DRM 3.58), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 3840x2160 b: Processor: AMD Ryzen 7 9800X3D 8-Core @ 6.22GHz (8 Cores / 16 Threads), Motherboard: ASRock X870E Taichi (3.12.AS02 BIOS), Chipset: AMD Device 14d8, Memory: 2 x 16GB DDR5-6000MT/s F5-6000J2836G16G, Disk: Western Digital WD_BLACK SN850X 2000GB, Graphics: AMD Radeon PRO W7500 8GB, Audio: AMD Navi 31 HDMI/DP, Monitor: DELL U2723QE, Network: Realtek Device 8126 + MEDIATEK Device 0717 OS: Ubuntu 24.04, Kernel: 6.8.0-49-generic (x86_64), Desktop: GNOME Shell 46.0, Display Server: X Server 1.21.1.11 + Wayland, OpenGL: 4.6 Mesa 24.2.0-devel (LLVM 18.1.7 DRM 3.58), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 3840x2160 AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB: Processor: AMD Ryzen 7 9800X3D 8-Core @ 6.22GHz (8 Cores / 16 Threads), Motherboard: ASRock X870E Taichi (3.12.AS02 BIOS), Chipset: AMD Device 14d8, Memory: 2 x 16GB DDR5-6000MT/s F5-6000J2836G16G, Disk: Western Digital WD_BLACK SN850X 2000GB, Graphics: AMD Radeon PRO W7500 8GB, Audio: AMD Navi 31 HDMI/DP, Monitor: DELL U2723QE, Network: Realtek Device 8126 + MEDIATEK Device 0717 OS: Ubuntu 24.04, Kernel: 6.8.0-49-generic (x86_64), Desktop: GNOME Shell 46.0, Display Server: X Server 1.21.1.11 + Wayland, OpenGL: 4.6 Mesa 24.2.0-devel (LLVM 18.1.7 DRM 3.58), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 3840x2160 c: Processor: AMD Ryzen 7 9800X3D 8-Core @ 6.22GHz (8 Cores / 16 Threads), Motherboard: ASRock X870E Taichi (3.12.AS02 BIOS), Chipset: AMD Device 14d8, Memory: 2 x 16GB DDR5-6000MT/s F5-6000J2836G16G, Disk: Western Digital WD_BLACK SN850X 2000GB, Graphics: AMD Radeon PRO W7500 8GB, Audio: AMD Navi 31 HDMI/DP, Monitor: DELL U2723QE, Network: Realtek Device 8126 + MEDIATEK Device 0717 OS: Ubuntu 24.04, Kernel: 6.8.0-49-generic (x86_64), Desktop: GNOME Shell 46.0, Display Server: X Server 1.21.1.11 + Wayland, OpenGL: 4.6 Mesa 24.2.0-devel (LLVM 18.1.7 DRM 3.58), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 3840x2160 Llamafile 0.8.16 Model: Llama-3.2-3B-Instruct.Q6_K - Test: Text Generation 16 Tokens Per Second > Higher Is Better a ..................................................... 21.47 |================ b ..................................................... 20.78 |=============== AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 20.77 |=============== c ..................................................... 20.79 |=============== Llamafile 0.8.16 Model: Llama-3.2-3B-Instruct.Q6_K - Test: Text Generation 128 Tokens Per Second > Higher Is Better a ..................................................... 21.42 |================ b ..................................................... 21.42 |================ AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 21.43 |================ c ..................................................... 21.43 |================ Llamafile 0.8.16 Model: Llama-3.2-3B-Instruct.Q6_K - Test: Prompt Processing 256 Tokens Per Second > Higher Is Better a ..................................................... 4096 |================= b ..................................................... 4096 |================= AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 4096 |================= c ..................................................... 4096 |================= Llamafile 0.8.16 Model: Llama-3.2-3B-Instruct.Q6_K - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a ..................................................... 8192 |================= b ..................................................... 8192 |================= AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 8192 |================= c ..................................................... 8192 |================= Llamafile 0.8.16 Model: TinyLlama-1.1B-Chat-v1.0.BF16 - Test: Text Generation 16 Tokens Per Second > Higher Is Better a ..................................................... 27.54 |================ b ..................................................... 26.28 |=============== AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 26.42 |=============== c ..................................................... 26.51 |=============== Llamafile 0.8.16 Model: Llama-3.2-3B-Instruct.Q6_K - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a ..................................................... 16384 |================ b ..................................................... 16384 |================ AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 16384 |================ c ..................................................... 16384 |================ Llamafile 0.8.16 Model: Llama-3.2-3B-Instruct.Q6_K - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a ..................................................... 32768 |================ b ..................................................... 32768 |================ AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 32768 |================ c ..................................................... 32768 |================ Llamafile 0.8.16 Model: TinyLlama-1.1B-Chat-v1.0.BF16 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a ..................................................... 27.57 |================ b ..................................................... 27.51 |================ AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 27.55 |================ c ..................................................... 27.52 |================ Llamafile 0.8.16 Model: mistral-7b-instruct-v0.2.Q5_K_M - Test: Text Generation 16 Tokens Per Second > Higher Is Better a ..................................................... 11.19 |================ b ..................................................... 10.94 |================ AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 10.94 |================ c ..................................................... 10.95 |================ Llamafile 0.8.16 Model: TinyLlama-1.1B-Chat-v1.0.BF16 - Test: Prompt Processing 256 Tokens Per Second > Higher Is Better a ..................................................... 4096 |================= b ..................................................... 4096 |================= AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 4096 |================= c ..................................................... 4096 |================= Llamafile 0.8.16 Model: TinyLlama-1.1B-Chat-v1.0.BF16 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a ..................................................... 8192 |================= b ..................................................... 8192 |================= AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 8192 |================= c ..................................................... 8192 |================= Llamafile 0.8.16 Model: mistral-7b-instruct-v0.2.Q5_K_M - Test: Text Generation 128 Tokens Per Second > Higher Is Better a ..................................................... 11.31 |================ b ..................................................... 11.30 |================ AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 11.30 |================ c ..................................................... 11.30 |================ Llamafile 0.8.16 Model: wizardcoder-python-34b-v1.0.Q6_K - Test: Text Generation 16 Tokens Per Second > Higher Is Better a ..................................................... 2.09 |================= b ..................................................... 2.00 |================ AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 1.98 |================ c ..................................................... 1.96 |================ Llamafile 0.8.16 Model: TinyLlama-1.1B-Chat-v1.0.BF16 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a ..................................................... 16384 |================ b ..................................................... 16384 |================ AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 16384 |================ c ..................................................... 16384 |================ Llamafile 0.8.16 Model: TinyLlama-1.1B-Chat-v1.0.BF16 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a ..................................................... 32768 |================ b ..................................................... 32768 |================ AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 32768 |================ c ..................................................... 32768 |================ Llamafile 0.8.16 Model: wizardcoder-python-34b-v1.0.Q6_K - Test: Text Generation 128 Tokens Per Second > Higher Is Better a ..................................................... 2.1 |================== b ..................................................... 2.1 |================== AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 2.1 |================== c ..................................................... 2.1 |================== Llamafile 0.8.16 Model: mistral-7b-instruct-v0.2.Q5_K_M - Test: Prompt Processing 256 Tokens Per Second > Higher Is Better a ..................................................... 4096 |================= b ..................................................... 4096 |================= AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 4096 |================= c ..................................................... 4096 |================= Llamafile 0.8.16 Model: mistral-7b-instruct-v0.2.Q5_K_M - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a ..................................................... 8192 |================= b ..................................................... 8192 |================= AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 8192 |================= c ..................................................... 8192 |================= Llamafile 0.8.16 Model: mistral-7b-instruct-v0.2.Q5_K_M - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a ..................................................... 16384 |================ b ..................................................... 16384 |================ AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 16384 |================ c ..................................................... 16384 |================ Llamafile 0.8.16 Model: mistral-7b-instruct-v0.2.Q5_K_M - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a ..................................................... 32768 |================ b ..................................................... 32768 |================ AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 32768 |================ c ..................................................... 32768 |================ Llamafile 0.8.16 Model: wizardcoder-python-34b-v1.0.Q6_K - Test: Prompt Processing 256 Tokens Per Second > Higher Is Better a ..................................................... 1536 |================= b ..................................................... 1536 |================= AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 1536 |================= c ..................................................... 1536 |================= Llamafile 0.8.16 Model: wizardcoder-python-34b-v1.0.Q6_K - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a ..................................................... 3072 |================= b ..................................................... 3072 |================= AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 3072 |================= c ..................................................... 3072 |================= Llamafile 0.8.16 Model: wizardcoder-python-34b-v1.0.Q6_K - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a ..................................................... 6144 |================= b ..................................................... 6144 |================= AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 6144 |================= c ..................................................... 6144 |================= Llamafile 0.8.16 Model: wizardcoder-python-34b-v1.0.Q6_K - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a ..................................................... 12288 |================ b ..................................................... 12288 |================ AMD Ryzen 7 9800X3D 8-Core - AMD Radeon PRO W7500 8GB . 12288 |================ c ..................................................... 12288 |================