ncnn llama AMD Ryzen Threadripper 7980X 64-Cores testing with a System76 Thelio Major (FA Z5 BIOS) and AMD Radeon RX 6700 XT 12GB on Ubuntu 24.04 via the Phoronix Test Suite. a: Processor: AMD Ryzen Threadripper 7980X 64-Cores @ 5.37GHz (64 Cores / 128 Threads), Motherboard: System76 Thelio Major (FA Z5 BIOS), Chipset: AMD Device 14a4, Memory: 4 x 32GB DDR5-4800MT/s Micron MTC20F1045S1RC48BA2, Disk: 1000GB CT1000T700SSD5, Graphics: AMD Radeon RX 6700 XT 12GB, Audio: AMD Device 14cc, Monitor: DELL P2415Q, Network: Aquantia AQC113C NBase-T/IEEE + Realtek RTL8125 2.5GbE + Intel Wi-Fi 6E OS: Ubuntu 24.04, Kernel: 6.12.3-061203-generic (x86_64), Desktop: GNOME Shell 46.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.0.9-0ubuntu0.2 (LLVM 17.0.6 DRM 3.59), Compiler: GCC 13.3.0, File-System: ext4, Screen Resolution: 1920x1200 b: Processor: AMD Ryzen Threadripper 7980X 64-Cores @ 5.37GHz (64 Cores / 128 Threads), Motherboard: System76 Thelio Major (FA Z5 BIOS), Chipset: AMD Device 14a4, Memory: 4 x 32GB DDR5-4800MT/s Micron MTC20F1045S1RC48BA2, Disk: 1000GB CT1000T700SSD5, Graphics: AMD Radeon RX 6700 XT 12GB, Audio: AMD Device 14cc, Monitor: DELL P2415Q, Network: Aquantia AQC113C NBase-T/IEEE + Realtek RTL8125 2.5GbE + Intel Wi-Fi 6E OS: Ubuntu 24.04, Kernel: 6.12.3-061203-generic (x86_64), Desktop: GNOME Shell 46.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.0.9-0ubuntu0.2 (LLVM 17.0.6 DRM 3.59), Compiler: GCC 13.3.0, File-System: ext4, Screen Resolution: 1920x1200 c: Processor: AMD Ryzen Threadripper 7980X 64-Cores @ 5.37GHz (64 Cores / 128 Threads), Motherboard: System76 Thelio Major (FA Z5 BIOS), Chipset: AMD Device 14a4, Memory: 4 x 32GB DDR5-4800MT/s Micron MTC20F1045S1RC48BA2, Disk: 1000GB CT1000T700SSD5, Graphics: AMD Radeon RX 6700 XT 12GB, Audio: AMD Device 14cc, Monitor: DELL P2415Q, Network: Aquantia AQC113C NBase-T/IEEE + Realtek RTL8125 2.5GbE + Intel Wi-Fi 6E OS: Ubuntu 24.04, Kernel: 6.12.3-061203-generic (x86_64), Desktop: GNOME Shell 46.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.0.9-0ubuntu0.2 (LLVM 17.0.6 DRM 3.59), Compiler: GCC 13.3.0, File-System: ext4, Screen Resolution: 1920x1200 d: Processor: AMD Ryzen Threadripper 7980X 64-Cores @ 5.37GHz (64 Cores / 128 Threads), Motherboard: System76 Thelio Major (FA Z5 BIOS), Chipset: AMD Device 14a4, Memory: 4 x 32GB DDR5-4800MT/s Micron MTC20F1045S1RC48BA2, Disk: 1000GB CT1000T700SSD5, Graphics: AMD Radeon RX 6700 XT 12GB, Audio: AMD Device 14cc, Monitor: DELL P2415Q, Network: Aquantia AQC113C NBase-T/IEEE + Realtek RTL8125 2.5GbE + Intel Wi-Fi 6E OS: Ubuntu 24.04, Kernel: 6.12.3-061203-generic (x86_64), Desktop: GNOME Shell 46.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.0.9-0ubuntu0.2 (LLVM 17.0.6 DRM 3.59), Compiler: GCC 13.3.0, File-System: ext4, Screen Resolution: 1920x1200 NCNN 20241226 Target: CPU - Model: mobilenet ms < Lower Is Better a . 16.27 |================================================================= b . 16.04 |================================================================ c . 15.88 |=============================================================== d . 17.05 |==================================================================== NCNN 20241226 Target: CPU-v2-v2 - Model: mobilenet-v2 ms < Lower Is Better a . 7.94 |===================================================================== b . 7.80 |==================================================================== c . 7.84 |==================================================================== d . 7.83 |==================================================================== NCNN 20241226 Target: CPU-v3-v3 - Model: mobilenet-v3 ms < Lower Is Better a . 8.57 |===================================================================== b . 8.32 |=================================================================== c . 8.55 |===================================================================== d . 8.60 |===================================================================== NCNN 20241226 Target: CPU - Model: shufflenet-v2 ms < Lower Is Better a . 10.62 |=================================================================== b . 10.57 |=================================================================== c . 10.51 |=================================================================== d . 10.71 |==================================================================== NCNN 20241226 Target: CPU - Model: mnasnet ms < Lower Is Better a . 7.85 |===================================================================== b . 7.80 |==================================================================== c . 7.77 |==================================================================== d . 7.87 |===================================================================== NCNN 20241226 Target: CPU - Model: efficientnet-b0 ms < Lower Is Better a . 11.36 |==================================================================== b . 11.08 |================================================================== c . 11.27 |=================================================================== d . 11.32 |==================================================================== NCNN 20241226 Target: CPU - Model: blazeface ms < Lower Is Better a . 4.32 |================================================================= b . 4.31 |================================================================ c . 4.28 |================================================================ d . 4.62 |===================================================================== NCNN 20241226 Target: CPU - Model: googlenet ms < Lower Is Better a . 15.39 |==================================================================== b . 15.01 |================================================================== c . 15.27 |=================================================================== d . 15.32 |==================================================================== NCNN 20241226 Target: CPU - Model: vgg16 ms < Lower Is Better a . 31.59 |=================================================================== b . 30.24 |================================================================ c . 31.39 |================================================================== d . 32.29 |==================================================================== NCNN 20241226 Target: CPU - Model: resnet18 ms < Lower Is Better a . 8.51 |==================================================================== b . 8.40 |=================================================================== c . 8.44 |==================================================================== d . 8.59 |===================================================================== NCNN 20241226 Target: CPU - Model: alexnet ms < Lower Is Better a . 4.71 |===================================================================== b . 4.34 |================================================================ c . 4.37 |================================================================ d . 4.44 |================================================================= NCNN 20241226 Target: CPU - Model: resnet50 ms < Lower Is Better a . 15.26 |=============================================================== b . 14.86 |============================================================= c . 14.78 |============================================================= d . 16.45 |==================================================================== NCNN 20241226 Target: CPUv2-yolov3v2-yolov3 - Model: mobilenetv2-yolov3 ms < Lower Is Better a . 16.27 |================================================================= b . 16.04 |================================================================ c . 15.88 |=============================================================== d . 17.05 |==================================================================== NCNN 20241226 Target: CPU - Model: yolov4-tiny ms < Lower Is Better a . 25.05 |=================================================================== b . 24.59 |================================================================== c . 24.92 |=================================================================== d . 25.31 |==================================================================== NCNN 20241226 Target: CPU - Model: squeezenet_ssd ms < Lower Is Better a . 18.31 |================================================================= b . 17.54 |============================================================== c . 18.04 |================================================================ d . 19.18 |==================================================================== NCNN 20241226 Target: CPU - Model: regnety_400m ms < Lower Is Better a . 33.07 |==================================================================== b . 33.24 |==================================================================== c . 32.82 |=================================================================== d . 33.20 |==================================================================== NCNN 20241226 Target: CPU - Model: vision_transformer ms < Lower Is Better a . 43.49 |==================================================================== b . 40.76 |================================================================ c . 43.02 |=================================================================== d . 42.14 |================================================================== NCNN 20241226 Target: CPU - Model: FastestDet ms < Lower Is Better a . 12.72 |=================================================================== b . 12.92 |==================================================================== c . 12.78 |=================================================================== d . 12.23 |================================================================ NCNN 20241226 Target: Vulkan GPU - Model: mobilenet ms < Lower Is Better a . 15.53 |================================================================= b . 16.01 |=================================================================== c . 15.90 |================================================================== d . 16.27 |==================================================================== NCNN 20241226 Target: Vulkan GPU-v2-v2 - Model: mobilenet-v2 ms < Lower Is Better a . 7.65 |=================================================================== b . 7.84 |==================================================================== c . 7.86 |===================================================================== d . 7.91 |===================================================================== NCNN 20241226 Target: Vulkan GPU-v3-v3 - Model: mobilenet-v3 ms < Lower Is Better a . 8.47 |==================================================================== b . 8.08 |================================================================= c . 8.55 |===================================================================== d . 8.53 |===================================================================== NCNN 20241226 Target: Vulkan GPU - Model: shufflenet-v2 ms < Lower Is Better a . 10.64 |=================================================================== b . 10.31 |================================================================= c . 10.69 |==================================================================== d . 10.75 |==================================================================== NCNN 20241226 Target: Vulkan GPU - Model: mnasnet ms < Lower Is Better a . 7.84 |===================================================================== b . 7.21 |=============================================================== c . 7.83 |===================================================================== d . 7.81 |===================================================================== NCNN 20241226 Target: Vulkan GPU - Model: efficientnet-b0 ms < Lower Is Better a . 11.12 |=================================================================== b . 10.71 |================================================================= c . 11.27 |==================================================================== d . 11.24 |==================================================================== NCNN 20241226 Target: Vulkan GPU - Model: blazeface ms < Lower Is Better a . 4.25 |==================================================================== b . 4.16 |=================================================================== c . 4.29 |===================================================================== d . 4.28 |===================================================================== NCNN 20241226 Target: Vulkan GPU - Model: googlenet ms < Lower Is Better a . 15.13 |=================================================================== b . 14.79 |================================================================== c . 15.31 |==================================================================== d . 15.32 |==================================================================== NCNN 20241226 Target: Vulkan GPU - Model: vgg16 ms < Lower Is Better a . 30.77 |================================================================== b . 30.24 |================================================================= c . 31.58 |==================================================================== d . 30.64 |================================================================== NCNN 20241226 Target: Vulkan GPU - Model: resnet18 ms < Lower Is Better a . 8.41 |==================================================================== b . 8.49 |===================================================================== c . 8.40 |==================================================================== d . 8.42 |==================================================================== NCNN 20241226 Target: Vulkan GPU - Model: alexnet ms < Lower Is Better a . 4.35 |===================================================================== b . 4.30 |==================================================================== c . 4.38 |===================================================================== d . 4.34 |==================================================================== NCNN 20241226 Target: Vulkan GPU - Model: resnet50 ms < Lower Is Better a . 14.67 |================================================================== b . 15.05 |=================================================================== c . 14.70 |================================================================== d . 15.18 |==================================================================== NCNN 20241226 Target: Vulkan GPUv2-yolov3v2-yolov3 - Model: mobilenetv2-yolov3 ms < Lower Is Better a . 15.53 |================================================================= b . 16.01 |=================================================================== c . 15.90 |================================================================== d . 16.27 |==================================================================== NCNN 20241226 Target: Vulkan GPU - Model: yolov4-tiny ms < Lower Is Better a . 24.20 |================================================================== b . 23.73 |================================================================= c . 24.89 |==================================================================== d . 24.50 |=================================================================== NCNN 20241226 Target: Vulkan GPU - Model: squeezenet_ssd ms < Lower Is Better a . 17.62 |================================================================== b . 16.84 |=============================================================== c . 18.05 |==================================================================== d . 17.69 |=================================================================== NCNN 20241226 Target: Vulkan GPU - Model: regnety_400m ms < Lower Is Better a . 32.46 |================================================================== b . 32.64 |=================================================================== c . 33.30 |==================================================================== d . 32.83 |=================================================================== NCNN 20241226 Target: Vulkan GPU - Model: vision_transformer ms < Lower Is Better a . 42.37 |==================================================================== b . 41.61 |=================================================================== c . 42.21 |==================================================================== d . 41.14 |================================================================== NCNN 20241226 Target: Vulkan GPU - Model: FastestDet ms < Lower Is Better a . 12.74 |==================================================================== b . 12.51 |=================================================================== c . 11.19 |============================================================ d . 12.63 |=================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 15.78 |==================================================================== b . 15.65 |=================================================================== c . 15.81 |==================================================================== d . 15.79 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 68.70 |================================================================== b . 70.27 |==================================================================== c . 69.09 |=================================================================== d . 70.54 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 70.15 |==================================================================== b . 70.02 |==================================================================== c . 70.15 |==================================================================== d . 69.11 |=================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 69.77 |==================================================================== b . 68.58 |=================================================================== c . 69.22 |=================================================================== d . 69.96 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 16.60 |==================================================================== b . 16.66 |==================================================================== c . 16.52 |=================================================================== d . 16.67 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 70.76 |==================================================================== b . 68.65 |================================================================== c . 68.65 |================================================================== d . 69.66 |=================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 69.79 |==================================================================== b . 70.30 |==================================================================== c . 69.62 |=================================================================== d . 68.76 |=================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 69.39 |=================================================================== b . 69.99 |==================================================================== c . 69.54 |==================================================================== d . 69.74 |==================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 75.57 |==================================================================== b . 75.40 |==================================================================== c . 75.88 |==================================================================== d . 75.13 |=================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 268.35 |=============================================================== b . 272.33 |================================================================ c . 283.43 |=================================================================== d . 283.50 |=================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 275.79 |================================================================== b . 278.00 |=================================================================== c . 268.94 |================================================================= d . 277.51 |=================================================================== Llama.cpp b4397 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 250.90 |================================================================ b . 242.95 |============================================================== c . 250.67 |================================================================ d . 260.63 |===================================================================