ncnn llama ryzen AMD Ryzen 7 7840HS testing with a Framework Laptop 16 (AMD Ryzen 7040 ) FRANMZCP07 (03.01 BIOS) and AMD Radeon 780M 512MB on Ubuntu 24.04 via the Phoronix Test Suite. a: Processor: AMD Ryzen 7 7840HS @ 5.29GHz (8 Cores / 16 Threads), Motherboard: Framework Laptop 16 (AMD Ryzen 7040 ) FRANMZCP07 (03.01 BIOS), Chipset: AMD Device 14e8, Memory: 2 x 8GB DDR5-5600MT/s A-DATA AD5S56008G-B, Disk: 512GB Western Digital PC SN810 SDCPNRY-512G, Graphics: AMD Radeon 780M 512MB, Audio: AMD Navi 31 HDMI/DP, Network: MEDIATEK MT7922 802.11ax PCI OS: Ubuntu 24.04, Kernel: 6.8.0-49-generic (x86_64), Desktop: GNOME Shell 46.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.2~git2406200600.0ac0fb~oibaf~n (git-0ac0fbc 2024-06-20 noble-oibaf-ppa) (LLVM 17.0.6 DRM 3.57), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 2560x1600 b: Processor: AMD Ryzen 7 7840HS @ 5.29GHz (8 Cores / 16 Threads), Motherboard: Framework Laptop 16 (AMD Ryzen 7040 ) FRANMZCP07 (03.01 BIOS), Chipset: AMD Device 14e8, Memory: 2 x 8GB DDR5-5600MT/s A-DATA AD5S56008G-B, Disk: 512GB Western Digital PC SN810 SDCPNRY-512G, Graphics: AMD Radeon 780M 512MB, Audio: AMD Navi 31 HDMI/DP, Network: MEDIATEK MT7922 802.11ax PCI OS: Ubuntu 24.04, Kernel: 6.8.0-49-generic (x86_64), Desktop: GNOME Shell 46.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.2~git2406200600.0ac0fb~oibaf~n (git-0ac0fbc 2024-06-20 noble-oibaf-ppa) (LLVM 17.0.6 DRM 3.57), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 2560x1600 c: Processor: AMD Ryzen 7 7840HS @ 5.29GHz (8 Cores / 16 Threads), Motherboard: Framework Laptop 16 (AMD Ryzen 7040 ) FRANMZCP07 (03.01 BIOS), Chipset: AMD Device 14e8, Memory: 2 x 8GB DDR5-5600MT/s A-DATA AD5S56008G-B, Disk: 512GB Western Digital PC SN810 SDCPNRY-512G, Graphics: AMD Radeon 780M 512MB, Audio: AMD Navi 31 HDMI/DP, Network: MEDIATEK MT7922 802.11ax PCI OS: Ubuntu 24.04, Kernel: 6.8.0-49-generic (x86_64), Desktop: GNOME Shell 46.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.2~git2406200600.0ac0fb~oibaf~n (git-0ac0fbc 2024-06-20 noble-oibaf-ppa) (LLVM 17.0.6 DRM 3.57), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 2560x1600 NCNN 20241226 Target: CPU - Model: mobilenet ms < Lower Is Better a . 10.42 |==================================================================== b . 9.95 |================================================================= c . 9.90 |================================================================= NCNN 20241226 Target: CPU-v2-v2 - Model: mobilenet-v2 ms < Lower Is Better a . 3.01 |==================================================== b . 4.01 |===================================================================== c . 4.02 |===================================================================== NCNN 20241226 Target: CPU-v3-v3 - Model: mobilenet-v3 ms < Lower Is Better a . 2.96 |===================================================================== b . 2.75 |================================================================ c . 2.75 |================================================================ NCNN 20241226 Target: CPU - Model: shufflenet-v2 ms < Lower Is Better a . 2.50 |===================================================================== b . 2.38 |================================================================== c . 2.37 |================================================================= NCNN 20241226 Target: CPU - Model: mnasnet ms < Lower Is Better a . 2.81 |===================================================================== b . 2.55 |=============================================================== c . 2.55 |=============================================================== NCNN 20241226 Target: CPU - Model: efficientnet-b0 ms < Lower Is Better a . 4.29 |===================================================================== b . 4.03 |================================================================= c . 4.02 |================================================================= NCNN 20241226 Target: CPU - Model: blazeface ms < Lower Is Better a . 0.85 |===================================================================== b . 0.74 |============================================================ c . 0.74 |============================================================ NCNN 20241226 Target: CPU - Model: googlenet ms < Lower Is Better a . 8.57 |===================================================================== b . 7.63 |============================================================= c . 7.61 |============================================================= NCNN 20241226 Target: CPU - Model: vgg16 ms < Lower Is Better a . 41.13 |==================================================================== b . 39.50 |================================================================= c . 39.64 |================================================================== NCNN 20241226 Target: CPU - Model: resnet18 ms < Lower Is Better a . 6.44 |===================================================================== b . 5.52 |=========================================================== c . 5.47 |=========================================================== NCNN 20241226 Target: CPU - Model: alexnet ms < Lower Is Better a . 5.54 |===================================================================== b . 5.23 |================================================================= c . 5.23 |================================================================= NCNN 20241226 Target: CPU - Model: resnet50 ms < Lower Is Better a . 13.96 |==================================================================== b . 12.71 |============================================================== c . 12.72 |============================================================== NCNN 20241226 Target: CPUv2-yolov3v2-yolov3 - Model: mobilenetv2-yolov3 ms < Lower Is Better a . 10.42 |==================================================================== b . 9.95 |================================================================= c . 9.90 |================================================================= NCNN 20241226 Target: CPU - Model: yolov4-tiny ms < Lower Is Better a . 16.69 |==================================================================== b . 15.58 |=============================================================== c . 15.47 |=============================================================== NCNN 20241226 Target: CPU - Model: squeezenet_ssd ms < Lower Is Better a . 7.21 |===================================================================== b . 6.83 |================================================================= c . 6.80 |================================================================= NCNN 20241226 Target: CPU - Model: regnety_400m ms < Lower Is Better a . 6.07 |===================================================================== b . 5.93 |=================================================================== c . 5.88 |=================================================================== NCNN 20241226 Target: CPU - Model: vision_transformer ms < Lower Is Better a . 59.71 |==================================================================== b . 59.95 |==================================================================== c . 59.94 |==================================================================== NCNN 20241226 Target: CPU - Model: FastestDet ms < Lower Is Better a . 2.67 |================================================================= b . 2.83 |===================================================================== c . 2.59 |=============================================================== NCNN 20241226 Target: Vulkan GPU - Model: mobilenet ms < Lower Is Better a . 10.45 |==================================================================== b . 9.89 |================================================================ c . 9.97 |================================================================= NCNN 20241226 Target: Vulkan GPU-v2-v2 - Model: mobilenet-v2 ms < Lower Is Better a . 3.03 |===================================================================== b . 2.85 |================================================================= c . 2.87 |================================================================= NCNN 20241226 Target: Vulkan GPU-v3-v3 - Model: mobilenet-v3 ms < Lower Is Better a . 2.98 |===================================================================== b . 2.77 |================================================================ c . 2.75 |================================================================ NCNN 20241226 Target: Vulkan GPU - Model: shufflenet-v2 ms < Lower Is Better a . 2.52 |===================================================================== b . 2.37 |================================================================= c . 2.36 |================================================================= NCNN 20241226 Target: Vulkan GPU - Model: mnasnet ms < Lower Is Better a . 2.81 |===================================================================== b . 2.56 |=============================================================== c . 2.59 |================================================================ NCNN 20241226 Target: Vulkan GPU - Model: efficientnet-b0 ms < Lower Is Better a . 4.33 |===================================================================== b . 4.02 |================================================================ c . 4.01 |================================================================ NCNN 20241226 Target: Vulkan GPU - Model: blazeface ms < Lower Is Better a . 0.79 |===================================================================== b . 0.74 |================================================================= c . 0.74 |================================================================= NCNN 20241226 Target: Vulkan GPU - Model: googlenet ms < Lower Is Better a . 8.76 |===================================================================== b . 7.50 |=========================================================== c . 7.44 |=========================================================== NCNN 20241226 Target: Vulkan GPU - Model: vgg16 ms < Lower Is Better a . 41.34 |=================================================================== b . 41.73 |==================================================================== c . 38.85 |=============================================================== NCNN 20241226 Target: Vulkan GPU - Model: resnet18 ms < Lower Is Better a . 6.51 |===================================================================== b . 4.91 |==================================================== c . 4.83 |=================================================== NCNN 20241226 Target: Vulkan GPU - Model: alexnet ms < Lower Is Better a . 5.56 |===================================================================== b . 4.87 |============================================================ c . 4.91 |============================================================= NCNN 20241226 Target: Vulkan GPU - Model: resnet50 ms < Lower Is Better a . 13.96 |============================================================== b . 15.32 |==================================================================== c . 12.19 |====================================================== NCNN 20241226 Target: Vulkan GPUv2-yolov3v2-yolov3 - Model: mobilenetv2-yolov3 ms < Lower Is Better a . 10.45 |==================================================================== b . 9.89 |================================================================ c . 9.97 |================================================================= NCNN 20241226 Target: Vulkan GPU - Model: yolov4-tiny ms < Lower Is Better a . 16.72 |==================================================================== b . 15.32 |============================================================== c . 15.11 |============================================================= NCNN 20241226 Target: Vulkan GPU - Model: squeezenet_ssd ms < Lower Is Better a . 7.31 |===================================================================== b . 6.96 |================================================================== c . 6.76 |================================================================ NCNN 20241226 Target: Vulkan GPU - Model: regnety_400m ms < Lower Is Better a . 6.09 |===================================================================== b . 5.86 |================================================================== c . 5.85 |================================================================== NCNN 20241226 Target: Vulkan GPU - Model: vision_transformer ms < Lower Is Better a . 60.18 |==================================================================== b . 59.62 |=================================================================== c . 59.85 |==================================================================== NCNN 20241226 Target: Vulkan GPU - Model: FastestDet ms < Lower Is Better a . 2.85 |==================================================================== b . 2.91 |===================================================================== c . 2.72 |================================================================ Llama.cpp b4397 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 7.17 |===================================================================== b . 7.21 |===================================================================== c . 7.21 |===================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 40.55 |==================================================================== b . 40.06 |=================================================================== c . 40.57 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 39.90 |==================================================================== b . 39.71 |=================================================================== c . 40.11 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 37.65 |================================================================== b . 38.54 |==================================================================== c . 38.74 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 7.60 |===================================================================== b . 7.60 |===================================================================== c . 7.59 |===================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 39.58 |================================================================== b . 40.20 |=================================================================== c . 40.56 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 39.59 |==================================================================== b . 39.54 |==================================================================== c . 39.49 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 38.27 |=================================================================== b . 38.59 |==================================================================== c . 38.50 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 53.66 |==================================================================== b . 53.55 |==================================================================== c . 53.55 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 165.04 |================================================================= b . 158.70 |============================================================== c . 170.43 |=================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 162.78 |================================================================== b . 160.59 |================================================================= c . 164.58 |=================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 148.11 |================================================================== b . 151.48 |=================================================================== c . 149.08 |==================================================================