adl ncnn llama Intel Core i7-1280P testing with a MSI Prestige 14Evo A12M MS-14C6 (E14C6IMS.115 BIOS) and MSI Intel ADL GT2 8GB on Ubuntu 24.10 via the Phoronix Test Suite. a: Processor: Intel Core i7-1280P @ 4.70GHz (14 Cores / 20 Threads), Motherboard: MSI Prestige 14Evo A12M MS-14C6 (E14C6IMS.115 BIOS), Chipset: Intel Alder Lake PCH, Memory: 8 x 2GB LPDDR4-4267MT/s SK Hynix H9HCNNNCPMMLXR-, Disk: 1024GB Micron_3400_MTFDKBA1T0TFH, Graphics: MSI Intel ADL GT2 8GB, Audio: Realtek ALC274, Network: Intel Alder Lake-P PCH CNVi WiFi OS: Ubuntu 24.10, Kernel: 6.11.0-rc6-phx (x86_64), Desktop: GNOME Shell 47.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.2.3-1ubuntu1, Compiler: GCC 14.2.0, File-System: ext4, Screen Resolution: 1920x1080 b: Processor: Intel Core i7-1280P @ 4.70GHz (14 Cores / 20 Threads), Motherboard: MSI Prestige 14Evo A12M MS-14C6 (E14C6IMS.115 BIOS), Chipset: Intel Alder Lake PCH, Memory: 8 x 2GB LPDDR4-4267MT/s SK Hynix H9HCNNNCPMMLXR-, Disk: 1024GB Micron_3400_MTFDKBA1T0TFH, Graphics: MSI Intel ADL GT2 8GB, Audio: Realtek ALC274, Network: Intel Alder Lake-P PCH CNVi WiFi OS: Ubuntu 24.10, Kernel: 6.11.0-rc6-phx (x86_64), Desktop: GNOME Shell 47.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.2.3-1ubuntu1, Compiler: GCC 14.2.0, File-System: ext4, Screen Resolution: 1920x1080 c: Processor: Intel Core i7-1280P @ 4.70GHz (14 Cores / 20 Threads), Motherboard: MSI Prestige 14Evo A12M MS-14C6 (E14C6IMS.115 BIOS), Chipset: Intel Alder Lake PCH, Memory: 8 x 2GB LPDDR4-4267MT/s SK Hynix H9HCNNNCPMMLXR-, Disk: 1024GB Micron_3400_MTFDKBA1T0TFH, Graphics: MSI Intel ADL GT2 8GB, Audio: Realtek ALC274, Network: Intel Alder Lake-P PCH CNVi WiFi OS: Ubuntu 24.10, Kernel: 6.11.0-rc6-phx (x86_64), Desktop: GNOME Shell 47.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.2.3-1ubuntu1, Compiler: GCC 14.2.0, File-System: ext4, Screen Resolution: 1920x1080 NCNN 20241226 Target: CPU - Model: mobilenet ms < Lower Is Better a . 17.19 |=================================================================== b . 17.38 |==================================================================== c . 17.00 |=================================================================== NCNN 20241226 Target: CPU-v2-v2 - Model: mobilenet-v2 ms < Lower Is Better a . 6.09 |===================================================================== b . 5.47 |============================================================== c . 6.03 |==================================================================== NCNN 20241226 Target: CPU-v3-v3 - Model: mobilenet-v3 ms < Lower Is Better a . 6.28 |===================================================================== b . 5.80 |================================================================ c . 5.46 |============================================================ NCNN 20241226 Target: CPU - Model: shufflenet-v2 ms < Lower Is Better a . 4.73 |========================================================= b . 5.07 |============================================================= c . 5.75 |===================================================================== NCNN 20241226 Target: CPU - Model: mnasnet ms < Lower Is Better a . 6.03 |===================================================================== b . 5.56 |================================================================ c . 5.75 |================================================================== NCNN 20241226 Target: CPU - Model: efficientnet-b0 ms < Lower Is Better a . 10.98 |==================================================================== b . 10.24 |=============================================================== c . 10.00 |============================================================== NCNN 20241226 Target: CPU - Model: blazeface ms < Lower Is Better a . 2.82 |===================================================================== b . 2.76 |==================================================================== c . 2.73 |=================================================================== NCNN 20241226 Target: CPU - Model: googlenet ms < Lower Is Better a . 12.29 |================================================================= b . 12.05 |================================================================ c . 12.79 |==================================================================== NCNN 20241226 Target: CPU - Model: vgg16 ms < Lower Is Better a . 33.18 |==================================================================== b . 33.15 |==================================================================== c . 33.02 |==================================================================== NCNN 20241226 Target: CPU - Model: resnet18 ms < Lower Is Better a . 7.50 |===================================================================== b . 7.32 |=================================================================== c . 7.54 |===================================================================== NCNN 20241226 Target: CPU - Model: alexnet ms < Lower Is Better a . 5.70 |==================================================================== b . 5.77 |===================================================================== c . 5.67 |==================================================================== NCNN 20241226 Target: CPU - Model: resnet50 ms < Lower Is Better a . 22.23 |==================================================================== b . 20.04 |============================================================= c . 18.05 |======================================================= NCNN 20241226 Target: CPUv2-yolov3v2-yolov3 - Model: mobilenetv2-yolov3 ms < Lower Is Better a . 17.19 |=================================================================== b . 17.38 |==================================================================== c . 17.00 |=================================================================== NCNN 20241226 Target: CPU - Model: yolov4-tiny ms < Lower Is Better a . 21.03 |=================================================================== b . 21.25 |==================================================================== c . 20.46 |================================================================= NCNN 20241226 Target: CPU - Model: squeezenet_ssd ms < Lower Is Better a . 13.15 |=================================================================== b . 13.36 |==================================================================== c . 11.96 |============================================================= NCNN 20241226 Target: CPU - Model: regnety_400m ms < Lower Is Better a . 26.21 |=================================================================== b . 24.49 |=============================================================== c . 26.64 |==================================================================== NCNN 20241226 Target: CPU - Model: vision_transformer ms < Lower Is Better a . 150.41 |=================================================================== b . 150.05 |=================================================================== c . 150.27 |=================================================================== NCNN 20241226 Target: CPU - Model: FastestDet ms < Lower Is Better a . 7.81 |===================================================================== b . 7.53 |=================================================================== c . 7.54 |=================================================================== NCNN 20241226 Target: Vulkan GPU - Model: mobilenet ms < Lower Is Better a . 17.06 |================================================================== b . 17.45 |==================================================================== c . 16.75 |================================================================= NCNN 20241226 Target: Vulkan GPU-v2-v2 - Model: mobilenet-v2 ms < Lower Is Better a . 6.08 |==================================================================== b . 6.18 |===================================================================== c . 6.19 |===================================================================== NCNN 20241226 Target: Vulkan GPU-v3-v3 - Model: mobilenet-v3 ms < Lower Is Better a . 5.89 |==================================================================== b . 5.95 |===================================================================== c . 5.75 |=================================================================== NCNN 20241226 Target: Vulkan GPU - Model: shufflenet-v2 ms < Lower Is Better a . 5.59 |==================================================================== b . 5.71 |===================================================================== c . 5.01 |============================================================= NCNN 20241226 Target: Vulkan GPU - Model: mnasnet ms < Lower Is Better a . 5.93 |================================================================== b . 6.16 |===================================================================== c . 5.95 |=================================================================== NCNN 20241226 Target: Vulkan GPU - Model: efficientnet-b0 ms < Lower Is Better a . 9.95 |============================================================== b . 10.57 |================================================================== c . 10.87 |==================================================================== NCNN 20241226 Target: Vulkan GPU - Model: blazeface ms < Lower Is Better a . 3.13 |===================================================================== b . 2.74 |============================================================ c . 2.70 |============================================================ NCNN 20241226 Target: Vulkan GPU - Model: googlenet ms < Lower Is Better a . 12.79 |==================================================================== b . 12.48 |================================================================== c . 12.38 |================================================================== NCNN 20241226 Target: Vulkan GPU - Model: vgg16 ms < Lower Is Better a . 38.91 |==================================================================== b . 37.77 |================================================================== c . 38.85 |==================================================================== NCNN 20241226 Target: Vulkan GPU - Model: resnet18 ms < Lower Is Better a . 7.36 |==================================================================== b . 7.46 |===================================================================== c . 7.35 |==================================================================== NCNN 20241226 Target: Vulkan GPU - Model: alexnet ms < Lower Is Better a . 5.79 |===================================================================== b . 5.79 |===================================================================== c . 5.79 |===================================================================== NCNN 20241226 Target: Vulkan GPU - Model: resnet50 ms < Lower Is Better a . 22.58 |==================================================================== b . 22.67 |==================================================================== c . 22.59 |==================================================================== NCNN 20241226 Target: Vulkan GPUv2-yolov3v2-yolov3 - Model: mobilenetv2-yolov3 ms < Lower Is Better a . 17.06 |================================================================== b . 17.45 |==================================================================== c . 16.75 |================================================================= NCNN 20241226 Target: Vulkan GPU - Model: yolov4-tiny ms < Lower Is Better a . 20.88 |=================================================================== b . 21.13 |==================================================================== c . 20.83 |=================================================================== NCNN 20241226 Target: Vulkan GPU - Model: squeezenet_ssd ms < Lower Is Better a . 13.28 |==================================================================== b . 13.17 |=================================================================== c . 13.15 |=================================================================== NCNN 20241226 Target: Vulkan GPU - Model: regnety_400m ms < Lower Is Better a . 25.72 |==================================================================== b . 25.83 |==================================================================== c . 24.97 |================================================================== NCNN 20241226 Target: Vulkan GPU - Model: vision_transformer ms < Lower Is Better a . 151.36 |=================================================================== b . 151.91 |=================================================================== c . 149.34 |================================================================== NCNN 20241226 Target: Vulkan GPU - Model: FastestDet ms < Lower Is Better a . 7.43 |=================================================================== b . 7.62 |===================================================================== c . 7.35 |=================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 6.62 |===================================================================== b . 6.61 |===================================================================== c . 6.64 |===================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 15.72 |=================================================================== b . 15.86 |==================================================================== c . 15.74 |=================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 15.53 |==================================================================== b . 15.45 |==================================================================== c . 15.42 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 15.07 |==================================================================== b . 15.03 |==================================================================== c . 15.06 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better b . 6.92 |===================================================================== c . 6.94 |===================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 15.76 |==================================================================== b . 15.75 |==================================================================== c . 15.75 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 15.56 |==================================================================== b . 15.57 |==================================================================== c . 15.54 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 15.11 |==================================================================== b . 15.10 |==================================================================== c . 15.09 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 29.32 |==================================================================== b . 29.36 |==================================================================== c . 28.77 |=================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 65.82 |==================================================================== b . 65.60 |==================================================================== c . 65.58 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 61.82 |==================================================================== b . 61.96 |==================================================================== c . 61.76 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 57.11 |==================================================================== b . 57.22 |==================================================================== c . 57.40 |====================================================================