ncnn llama arm

ARMv8 Neoverse-N1 testing with a System76 Thelio Astra (3.02 BIOS) and NVIDIA RTX A400/PCIe 4GB on Ubuntu 24.04 via the Phoronix Test Suite.

HTML result view exported from: https://openbenchmarking.org/result/2412300-PTS-NCNNLLAM46&rdt.

NCNN

Target: CPU - Model: mobilenet

NCNN

Target: CPU-v2-v2 - Model: mobilenet-v2

NCNN

Target: CPU-v3-v3 - Model: mobilenet-v3

NCNN

Target: CPU - Model: shufflenet-v2

NCNN

Target: CPU - Model: mnasnet

NCNN

Target: CPU - Model: efficientnet-b0

NCNN

Target: CPU - Model: blazeface

NCNN

Target: CPU - Model: googlenet

NCNN

Target: CPU - Model: vgg16

NCNN

Target: CPU - Model: resnet18

NCNN

Target: CPU - Model: alexnet

NCNN

Target: CPU - Model: resnet50

NCNN

Target: CPUv2-yolov3v2-yolov3 - Model: mobilenetv2-yolov3

NCNN

Target: CPU - Model: yolov4-tiny

NCNN

Target: CPU - Model: squeezenet_ssd

NCNN

Target: CPU - Model: regnety_400m

NCNN

Target: CPU - Model: vision_transformer

NCNN

Target: CPU - Model: FastestDet

NCNN

Target: Vulkan GPU - Model: mobilenet

NCNN

Target: Vulkan GPU-v2-v2 - Model: mobilenet-v2

NCNN

Target: Vulkan GPU-v3-v3 - Model: mobilenet-v3

NCNN

Target: Vulkan GPU - Model: shufflenet-v2

NCNN

Target: Vulkan GPU - Model: mnasnet

NCNN

Target: Vulkan GPU - Model: efficientnet-b0

NCNN

Target: Vulkan GPU - Model: blazeface

NCNN

Target: Vulkan GPU - Model: googlenet

NCNN

Target: Vulkan GPU - Model: vgg16

NCNN

Target: Vulkan GPU - Model: resnet18

NCNN

Target: Vulkan GPU - Model: alexnet

NCNN

Target: Vulkan GPU - Model: resnet50

NCNN

Target: Vulkan GPUv2-yolov3v2-yolov3 - Model: mobilenetv2-yolov3

NCNN

Target: Vulkan GPU - Model: yolov4-tiny

NCNN

Target: Vulkan GPU - Model: squeezenet_ssd

NCNN

Target: Vulkan GPU - Model: regnety_400m

NCNN

Target: Vulkan GPU - Model: vision_transformer

NCNN

Target: Vulkan GPU - Model: FastestDet

Llama.cpp

Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Text Generation 128

Llama.cpp

Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 512

Llama.cpp

Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 1024

Llama.cpp

Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 2048

Llama.cpp

Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Text Generation 128

Llama.cpp

Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 512

Llama.cpp

Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 1024

Llama.cpp

Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 2048

Llama.cpp

Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Text Generation 128

Llama.cpp

Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 512

Llama.cpp

Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 1024

Llama.cpp

Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 2048

Phoronix Test Suite v10.8.5