ncnn llama Intel Core Ultra 7 256V testing with a ASUS Zenbook S 14 UX5406SA_UX5406SA UX5406SA v1.0 (UX5406SA.300 BIOS) and ASUS Intel LNL 7GB on Ubuntu 24.10 via the Phoronix Test Suite. a: Processor: Intel Core Ultra 7 256V @ 4.70GHz (8 Cores), Motherboard: ASUS Zenbook S 14 UX5406SA_UX5406SA UX5406SA v1.0 (UX5406SA.300 BIOS), Chipset: Intel Device a87f, Memory: 8 x 2GB LPDDR5-8533MT/s Samsung, Disk: 1024GB Western Digital WD PC SN560 SDDPNQE-1T00-1102, Graphics: ASUS Intel LNL 7GB, Audio: Intel Lunar Lake-M HD Audio, Network: Intel Device a840 OS: Ubuntu 24.10, Kernel: 6.12.0-rc6-phx-drm-next (x86_64), Desktop: GNOME Shell 47.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 25.0~git2411250600.45c523~oibaf~o (git-45c5231 2024-11-25 oracular-oibaf-pp, OpenCL: OpenCL 3.0, Compiler: GCC 14.2.0, File-System: ext4, Screen Resolution: 2880x1800 b: Processor: Intel Core Ultra 7 256V @ 4.70GHz (8 Cores), Motherboard: ASUS Zenbook S 14 UX5406SA_UX5406SA UX5406SA v1.0 (UX5406SA.300 BIOS), Chipset: Intel Device a87f, Memory: 8 x 2GB LPDDR5-8533MT/s Samsung, Disk: 1024GB Western Digital WD PC SN560 SDDPNQE-1T00-1102, Graphics: ASUS Intel LNL 7GB, Audio: Intel Lunar Lake-M HD Audio, Network: Intel Device a840 OS: Ubuntu 24.10, Kernel: 6.12.0-rc6-phx-drm-next (x86_64), Desktop: GNOME Shell 47.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 25.0~git2411250600.45c523~oibaf~o (git-45c5231 2024-11-25 oracular-oibaf-pp, OpenCL: OpenCL 3.0, Compiler: GCC 14.2.0, File-System: ext4, Screen Resolution: 2880x1800 c: Processor: Intel Core Ultra 7 256V @ 4.70GHz (8 Cores), Motherboard: ASUS Zenbook S 14 UX5406SA_UX5406SA UX5406SA v1.0 (UX5406SA.300 BIOS), Chipset: Intel Device a87f, Memory: 8 x 2GB LPDDR5-8533MT/s Samsung, Disk: 1024GB Western Digital WD PC SN560 SDDPNQE-1T00-1102, Graphics: ASUS Intel LNL 7GB, Audio: Intel Lunar Lake-M HD Audio, Network: Intel Device a840 OS: Ubuntu 24.10, Kernel: 6.12.0-rc6-phx-drm-next (x86_64), Desktop: GNOME Shell 47.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 25.0~git2411250600.45c523~oibaf~o (git-45c5231 2024-11-25 oracular-oibaf-pp, OpenCL: OpenCL 3.0, Compiler: GCC 14.2.0, File-System: ext4, Screen Resolution: 2880x1800 d: Processor: Intel Core Ultra 7 256V @ 4.70GHz (8 Cores), Motherboard: ASUS Zenbook S 14 UX5406SA_UX5406SA UX5406SA v1.0 (UX5406SA.300 BIOS), Chipset: Intel Device a87f, Memory: 8 x 2GB LPDDR5-8533MT/s Samsung, Disk: 1024GB Western Digital WD PC SN560 SDDPNQE-1T00-1102, Graphics: ASUS Intel LNL 7GB, Audio: Intel Lunar Lake-M HD Audio, Network: Intel Device a840 OS: Ubuntu 24.10, Kernel: 6.12.0-rc6-phx-drm-next (x86_64), Desktop: GNOME Shell 47.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 25.0~git2411250600.45c523~oibaf~o (git-45c5231 2024-11-25 oracular-oibaf-pp, OpenCL: OpenCL 3.0, Compiler: GCC 14.2.0, File-System: ext4, Screen Resolution: 2880x1800 NCNN 20241226 Target: CPU - Model: mobilenet ms < Lower Is Better a . 14.14 |=================================================================== b . 13.59 |================================================================ c . 14.40 |==================================================================== d . 14.29 |=================================================================== NCNN 20241226 Target: CPU-v2-v2 - Model: mobilenet-v2 ms < Lower Is Better a . 7.01 |===================================================================== b . 5.00 |================================================= c . 4.90 |================================================ d . 4.98 |================================================= NCNN 20241226 Target: CPU-v3-v3 - Model: mobilenet-v3 ms < Lower Is Better a . 6.13 |===================================================================== b . 4.50 |=================================================== c . 4.43 |================================================== d . 4.53 |=================================================== NCNN 20241226 Target: CPU - Model: shufflenet-v2 ms < Lower Is Better a . 4.82 |===================================================================== b . 3.65 |==================================================== c . 3.64 |==================================================== d . 3.65 |==================================================== NCNN 20241226 Target: CPU - Model: mnasnet ms < Lower Is Better a . 6.33 |===================================================================== b . 4.48 |================================================= c . 4.49 |================================================= d . 4.52 |================================================= NCNN 20241226 Target: CPU - Model: efficientnet-b0 ms < Lower Is Better a . 11.16 |==================================================================== b . 7.99 |================================================= c . 7.84 |================================================ d . 7.95 |================================================ NCNN 20241226 Target: CPU - Model: blazeface ms < Lower Is Better a . 2.37 |===================================================================== b . 1.84 |====================================================== c . 1.88 |======================================================= d . 1.94 |======================================================== NCNN 20241226 Target: CPU - Model: googlenet ms < Lower Is Better a . 16.62 |==================================================================== b . 11.21 |============================================== c . 11.26 |============================================== d . 11.36 |============================================== NCNN 20241226 Target: CPU - Model: vgg16 ms < Lower Is Better a . 48.72 |==================================================================== b . 44.40 |============================================================== c . 44.85 |=============================================================== d . 44.32 |============================================================== NCNN 20241226 Target: CPU - Model: resnet18 ms < Lower Is Better a . 11.36 |==================================================================== b . 7.51 |============================================= c . 7.59 |============================================= d . 7.62 |============================================== NCNN 20241226 Target: CPU - Model: alexnet ms < Lower Is Better a . 8.39 |===================================================================== b . 5.86 |================================================ c . 6.12 |================================================== d . 6.06 |================================================== NCNN 20241226 Target: CPU - Model: resnet50 ms < Lower Is Better a . 30.49 |==================================================================== b . 18.96 |========================================== c . 19.25 |=========================================== d . 19.20 |=========================================== NCNN 20241226 Target: CPUv2-yolov3v2-yolov3 - Model: mobilenetv2-yolov3 ms < Lower Is Better a . 14.14 |=================================================================== b . 13.59 |================================================================ c . 14.40 |==================================================================== d . 14.29 |=================================================================== NCNN 20241226 Target: CPU - Model: yolov4-tiny ms < Lower Is Better a . 17.70 |=================================================================== b . 17.62 |=================================================================== c . 17.54 |=================================================================== d . 17.93 |==================================================================== NCNN 20241226 Target: CPU - Model: squeezenet_ssd ms < Lower Is Better a . 15.17 |==================================================================== b . 9.55 |=========================================== c . 9.66 |=========================================== d . 9.63 |=========================================== NCNN 20241226 Target: CPU - Model: regnety_400m ms < Lower Is Better a . 21.56 |==================================================================== b . 16.51 |==================================================== c . 16.90 |===================================================== d . 16.34 |==================================================== NCNN 20241226 Target: CPU - Model: vision_transformer ms < Lower Is Better a . 199.02 |=================================================================== b . 199.84 |=================================================================== c . 197.49 |================================================================== d . 199.30 |=================================================================== NCNN 20241226 Target: CPU - Model: FastestDet ms < Lower Is Better a . 4.93 |===================================================================== b . 4.69 |================================================================== c . 4.68 |================================================================= d . 4.94 |===================================================================== NCNN 20241226 Target: Vulkan GPU - Model: mobilenet ms < Lower Is Better a . 13.99 |================================================================== b . 14.35 |==================================================================== c . 13.88 |================================================================== d . 13.63 |================================================================= NCNN 20241226 Target: Vulkan GPU-v2-v2 - Model: mobilenet-v2 ms < Lower Is Better a . 4.75 |================================================================== b . 4.87 |=================================================================== c . 5.00 |===================================================================== d . 4.96 |==================================================================== NCNN 20241226 Target: Vulkan GPU-v3-v3 - Model: mobilenet-v3 ms < Lower Is Better a . 4.51 |===================================================================== b . 4.39 |=================================================================== c . 4.53 |===================================================================== d . 4.49 |==================================================================== NCNN 20241226 Target: Vulkan GPU - Model: shufflenet-v2 ms < Lower Is Better a . 3.65 |===================================================================== b . 3.66 |===================================================================== c . 3.65 |===================================================================== d . 3.66 |===================================================================== NCNN 20241226 Target: Vulkan GPU - Model: mnasnet ms < Lower Is Better a . 4.47 |===================================================================== b . 4.37 |=================================================================== c . 4.40 |==================================================================== d . 4.42 |==================================================================== NCNN 20241226 Target: Vulkan GPU - Model: efficientnet-b0 ms < Lower Is Better a . 7.92 |==================================================================== b . 7.99 |===================================================================== c . 7.87 |==================================================================== d . 7.98 |===================================================================== NCNN 20241226 Target: Vulkan GPU - Model: blazeface ms < Lower Is Better a . 1.92 |===================================================================== b . 1.92 |===================================================================== c . 1.92 |===================================================================== d . 1.91 |===================================================================== NCNN 20241226 Target: Vulkan GPU - Model: googlenet ms < Lower Is Better a . 11.25 |==================================================================== b . 11.28 |==================================================================== c . 11.27 |==================================================================== d . 11.25 |==================================================================== NCNN 20241226 Target: Vulkan GPU - Model: vgg16 ms < Lower Is Better a . 44.95 |==================================================================== b . 44.61 |=================================================================== c . 44.65 |=================================================================== d . 45.04 |==================================================================== NCNN 20241226 Target: Vulkan GPU - Model: resnet18 ms < Lower Is Better a . 7.56 |==================================================================== b . 7.65 |===================================================================== c . 7.55 |==================================================================== d . 7.56 |==================================================================== NCNN 20241226 Target: Vulkan GPU - Model: alexnet ms < Lower Is Better a . 5.91 |================================================================== b . 6.17 |===================================================================== c . 5.91 |================================================================== d . 5.94 |================================================================== NCNN 20241226 Target: Vulkan GPU - Model: resnet50 ms < Lower Is Better a . 19.10 |=================================================================== b . 19.26 |==================================================================== c . 19.06 |=================================================================== d . 19.06 |=================================================================== NCNN 20241226 Target: Vulkan GPUv2-yolov3v2-yolov3 - Model: mobilenetv2-yolov3 ms < Lower Is Better a . 13.99 |================================================================== b . 14.35 |==================================================================== c . 13.88 |================================================================== d . 13.63 |================================================================= NCNN 20241226 Target: Vulkan GPU - Model: yolov4-tiny ms < Lower Is Better a . 17.64 |=================================================================== b . 17.74 |==================================================================== c . 17.51 |=================================================================== d . 17.84 |==================================================================== NCNN 20241226 Target: Vulkan GPU - Model: squeezenet_ssd ms < Lower Is Better a . 9.57 |==================================================================== b . 9.77 |===================================================================== c . 9.66 |==================================================================== d . 9.70 |===================================================================== NCNN 20241226 Target: Vulkan GPU - Model: regnety_400m ms < Lower Is Better a . 16.59 |=================================================================== b . 16.73 |==================================================================== c . 16.39 |=================================================================== d . 16.25 |================================================================== NCNN 20241226 Target: Vulkan GPU - Model: vision_transformer ms < Lower Is Better a . 196.62 |================================================================= b . 200.12 |================================================================== c . 202.37 |=================================================================== d . 198.73 |================================================================== NCNN 20241226 Target: Vulkan GPU - Model: FastestDet ms < Lower Is Better a . 4.82 |=================================================================== b . 4.95 |===================================================================== c . 4.81 |=================================================================== d . 4.63 |================================================================= Llama.cpp b4397 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 8.87 |===================================================================== b . 8.85 |===================================================================== c . 8.85 |===================================================================== d . 8.83 |===================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 28.77 |==================================================================== b . 28.61 |==================================================================== c . 28.43 |=================================================================== d . 28.51 |=================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 28.09 |==================================================================== b . 27.93 |==================================================================== c . 27.78 |=================================================================== d . 26.87 |================================================================= Llama.cpp b4397 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 27.16 |==================================================================== b . 27.09 |==================================================================== c . 27.15 |==================================================================== d . 27.01 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 9.24 |===================================================================== b . 9.22 |===================================================================== c . 9.20 |===================================================================== d . 9.21 |===================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 28.70 |==================================================================== b . 28.30 |=================================================================== c . 28.54 |==================================================================== d . 28.50 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 28.00 |==================================================================== b . 27.90 |==================================================================== c . 26.25 |================================================================ d . 27.81 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 27.11 |==================================================================== b . 27.22 |==================================================================== c . 27.19 |==================================================================== d . 27.26 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 38.35 |==================================================================== b . 38.35 |==================================================================== c . 38.43 |==================================================================== d . 37.64 |=================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 61.73 |=================================================================== b . 61.34 |=================================================================== c . 60.85 |================================================================== d . 62.31 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 60.83 |=================================================================== b . 61.49 |==================================================================== c . 54.08 |============================================================ d . 54.39 |============================================================ Llama.cpp b4397 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 58.73 |================================================================= b . 61.40 |==================================================================== c . 56.79 |=============================================================== d . 57.28 |===============================================================