dddda AMD Ryzen AI 9 365 testing with a ASUS Zenbook S 16 UM5606WA_UM5606WA UM5606WA v1.0 (UM5606WA.308 BIOS) and AMD Radeon 512MB on Ubuntu 24.10 via the Phoronix Test Suite. a: Processor: AMD Ryzen AI 9 365 @ 4.31GHz (10 Cores / 20 Threads), Motherboard: ASUS Zenbook S 16 UM5606WA_UM5606WA UM5606WA v1.0 (UM5606WA.308 BIOS), Chipset: AMD Device 1507, Memory: 4 x 6GB LPDDR5-7500MT/s Micron MT62F1536M32D4DS-026, Disk: 1024GB MTFDKBA1T0QFM-1BD1AABGB, Graphics: AMD Radeon 512MB, Audio: AMD Rembrandt Radeon HD Audio, Network: MEDIATEK Device 7925 OS: Ubuntu 24.10, Kernel: 6.12.0-rc7-phx-eraps (x86_64), Desktop: GNOME Shell 47.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.2.3-1ubuntu1 (LLVM 19.1.0 DRM 3.59), Compiler: GCC 14.2.0, File-System: ext4, Screen Resolution: 2880x1800 b: Processor: AMD Ryzen AI 9 365 @ 4.31GHz (10 Cores / 20 Threads), Motherboard: ASUS Zenbook S 16 UM5606WA_UM5606WA UM5606WA v1.0 (UM5606WA.308 BIOS), Chipset: AMD Device 1507, Memory: 4 x 6GB LPDDR5-7500MT/s Micron MT62F1536M32D4DS-026, Disk: 1024GB MTFDKBA1T0QFM-1BD1AABGB, Graphics: AMD Radeon 512MB, Audio: AMD Rembrandt Radeon HD Audio, Network: MEDIATEK Device 7925 OS: Ubuntu 24.10, Kernel: 6.12.0-rc7-phx-eraps (x86_64), Desktop: GNOME Shell 47.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.2.3-1ubuntu1 (LLVM 19.1.0 DRM 3.59), Compiler: GCC 14.2.0, File-System: ext4, Screen Resolution: 2880x1800 c: Processor: AMD Ryzen AI 9 365 @ 4.31GHz (10 Cores / 20 Threads), Motherboard: ASUS Zenbook S 16 UM5606WA_UM5606WA UM5606WA v1.0 (UM5606WA.308 BIOS), Chipset: AMD Device 1507, Memory: 4 x 6GB LPDDR5-7500MT/s Micron MT62F1536M32D4DS-026, Disk: 1024GB MTFDKBA1T0QFM-1BD1AABGB, Graphics: AMD Radeon 512MB, Audio: AMD Rembrandt Radeon HD Audio, Network: MEDIATEK Device 7925 OS: Ubuntu 24.10, Kernel: 6.12.0-rc7-phx-eraps (x86_64), Desktop: GNOME Shell 47.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.2.3-1ubuntu1 (LLVM 19.1.0 DRM 3.59), Compiler: GCC 14.2.0, File-System: ext4, Screen Resolution: 2880x1800 OpenVINO 2024.5 Model: Face Detection FP16 - Device: CPU FPS > Higher Is Better a . 4.61 |===================================================================== b . 4.59 |===================================================================== c . 4.52 |==================================================================== OpenVINO 2024.5 Model: Person Detection FP16 - Device: CPU FPS > Higher Is Better a . 43.95 |==================================================================== b . 43.60 |=================================================================== c . 43.73 |==================================================================== OpenVINO 2024.5 Model: Person Detection FP32 - Device: CPU FPS > Higher Is Better a . 44.07 |==================================================================== b . 43.86 |==================================================================== c . 43.31 |=================================================================== OpenVINO 2024.5 Model: Vehicle Detection FP16 - Device: CPU FPS > Higher Is Better a . 283.78 |============================================================= b . 314.25 |=================================================================== c . 312.32 |=================================================================== OpenVINO 2024.5 Model: Face Detection FP16-INT8 - Device: CPU FPS > Higher Is Better a . 8.54 |================================================================== b . 8.89 |===================================================================== c . 8.03 |============================================================== OpenVINO 2024.5 Model: Face Detection Retail FP16 - Device: CPU FPS > Higher Is Better a . 950.04 |============================================================= b . 1032.88 |================================================================== c . 939.90 |============================================================ OpenVINO 2024.5 Model: Road Segmentation ADAS FP16 - Device: CPU FPS > Higher Is Better a . 126.02 |============================================================== b . 136.70 |=================================================================== c . 133.18 |================================================================= OpenVINO 2024.5 Model: Vehicle Detection FP16-INT8 - Device: CPU FPS > Higher Is Better a . 467.17 |============================================================== b . 507.16 |=================================================================== c . 496.31 |================================================================== OpenVINO 2024.5 Model: Weld Porosity Detection FP16 - Device: CPU FPS > Higher Is Better a . 446.41 |============================================================== b . 481.79 |=================================================================== c . 458.73 |================================================================ OpenVINO 2024.5 Model: Face Detection Retail FP16-INT8 - Device: CPU FPS > Higher Is Better a . 1587.70 |============================================================= b . 1712.41 |================================================================== c . 1676.72 |================================================================= OpenVINO 2024.5 Model: Road Segmentation ADAS FP16-INT8 - Device: CPU FPS > Higher Is Better a . 170.05 |============================================================== b . 175.74 |================================================================= c . 182.39 |=================================================================== OpenVINO 2024.5 Model: Machine Translation EN To DE FP16 - Device: CPU FPS > Higher Is Better a . 48.99 |============================================================= b . 54.49 |==================================================================== c . 53.46 |=================================================================== OpenVINO 2024.5 Model: Weld Porosity Detection FP16-INT8 - Device: CPU FPS > Higher Is Better a . 886.77 |============================================================== b . 953.24 |=================================================================== c . 920.13 |================================================================= OpenVINO 2024.5 Model: Person Vehicle Bike Detection FP16 - Device: CPU FPS > Higher Is Better a . 437.20 |============================================================= b . 481.89 |=================================================================== c . 474.85 |================================================================== OpenVINO 2024.5 Model: Noise Suppression Poconet-Like FP16 - Device: CPU FPS > Higher Is Better a . 607.91 |============================================================= b . 664.33 |=================================================================== c . 661.16 |=================================================================== OpenVINO 2024.5 Model: Handwritten English Recognition FP16 - Device: CPU FPS > Higher Is Better a . 245.49 |============================================================== b . 266.88 |=================================================================== c . 252.75 |=============================================================== OpenVINO 2024.5 Model: Person Re-Identification Retail FP16 - Device: CPU FPS > Higher Is Better a . 540.95 |================================================================= b . 557.70 |=================================================================== c . 561.16 |=================================================================== OpenVINO 2024.5 Model: Age Gender Recognition Retail 0013 FP16 - Device: CPU FPS > Higher Is Better a . 12337.97 |============================================================== b . 12742.09 |================================================================ c . 12864.63 |================================================================= OpenVINO 2024.5 Model: Handwritten English Recognition FP16-INT8 - Device: CPU FPS > Higher Is Better a . 254.39 |============================================================ b . 282.93 |=================================================================== c . 278.10 |================================================================== OpenVINO 2024.5 Model: Age Gender Recognition Retail 0013 FP16-INT8 - Device: CPU FPS > Higher Is Better a . 17776.30 |============================================================= b . 18962.53 |================================================================= c . 18885.08 |================================================================= Llama.cpp b4154 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 10.23 |=================================================================== b . 10.28 |==================================================================== c . 10.31 |==================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 36.83 |============================================================== b . 40.10 |==================================================================== c . 38.45 |================================================================= Llama.cpp b4154 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 35.89 |=================================================================== b . 36.10 |==================================================================== c . 36.19 |==================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 33.24 |================================================================== b . 34.19 |==================================================================== c . 33.65 |=================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 10.81 |==================================================================== b . 10.83 |==================================================================== c . 10.83 |==================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 36.83 |============================================================= b . 40.74 |==================================================================== c . 38.90 |================================================================= Llama.cpp b4154 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 35.93 |=================================================================== b . 36.41 |==================================================================== c . 35.66 |=================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 33.64 |================================================================== b . 34.47 |==================================================================== c . 33.81 |=================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 61.76 |==================================================================== b . 60.43 |=================================================================== c . 61.36 |==================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 158.11 |=================================================================== b . 157.49 |=================================================================== c . 158.18 |=================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 147.47 |=================================================================== b . 142.57 |================================================================= c . 147.12 |=================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 126.06 |======================================================= b . 153.71 |=================================================================== c . 137.25 |============================================================ Llamafile 0.8.16 Model: Llama-3.2-3B-Instruct.Q6_K - Test: Text Generation 16 Tokens Per Second > Higher Is Better a . 26.51 |==================================================================== b . 22.91 |=========================================================== c . 22.72 |========================================================== Llamafile 0.8.16 Model: Llama-3.2-3B-Instruct.Q6_K - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 26.24 |=================================================================== b . 26.44 |==================================================================== c . 26.31 |==================================================================== Llamafile 0.8.16 Model: Llama-3.2-3B-Instruct.Q6_K - Test: Prompt Processing 256 Tokens Per Second > Higher Is Better a . 4096 |===================================================================== b . 4096 |===================================================================== c . 4096 |===================================================================== Llamafile 0.8.16 Model: Llama-3.2-3B-Instruct.Q6_K - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 8192 |===================================================================== b . 8192 |===================================================================== c . 8192 |===================================================================== Llamafile 0.8.16 Model: TinyLlama-1.1B-Chat-v1.0.BF16 - Test: Text Generation 16 Tokens Per Second > Higher Is Better a . 34.27 |==================================================================== b . 29.99 |============================================================ c . 30.14 |============================================================ Llamafile 0.8.16 Model: Llama-3.2-3B-Instruct.Q6_K - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 16384 |==================================================================== b . 16384 |==================================================================== c . 16384 |==================================================================== Llamafile 0.8.16 Model: Llama-3.2-3B-Instruct.Q6_K - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 32768 |==================================================================== b . 32768 |==================================================================== c . 32768 |==================================================================== Llamafile 0.8.16 Model: TinyLlama-1.1B-Chat-v1.0.BF16 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 33.98 |==================================================================== b . 33.50 |=================================================================== c . 33.65 |=================================================================== Llamafile 0.8.16 Model: mistral-7b-instruct-v0.2.Q5_K_M - Test: Text Generation 16 Tokens Per Second > Higher Is Better a . 13.95 |==================================================================== b . 11.99 |========================================================== c . 12.17 |=========================================================== Llamafile 0.8.16 Model: TinyLlama-1.1B-Chat-v1.0.BF16 - Test: Prompt Processing 256 Tokens Per Second > Higher Is Better a . 4096 |===================================================================== b . 4096 |===================================================================== c . 4096 |===================================================================== Llamafile 0.8.16 Model: TinyLlama-1.1B-Chat-v1.0.BF16 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 8192 |===================================================================== b . 8192 |===================================================================== c . 8192 |===================================================================== Llamafile 0.8.16 Model: mistral-7b-instruct-v0.2.Q5_K_M - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 13.69 |=================================================================== b . 13.85 |==================================================================== c . 13.89 |==================================================================== Llamafile 0.8.16 Model: wizardcoder-python-34b-v1.0.Q6_K - Test: Text Generation 16 Tokens Per Second > Higher Is Better a . 0.08 |===================================================================== b . 0.07 |============================================================ c . 0.07 |============================================================ Llamafile 0.8.16 Model: TinyLlama-1.1B-Chat-v1.0.BF16 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 16384 |==================================================================== b . 16384 |==================================================================== c . 16384 |==================================================================== Llamafile 0.8.16 Model: TinyLlama-1.1B-Chat-v1.0.BF16 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 32768 |==================================================================== b . 32768 |==================================================================== c . 32768 |==================================================================== OpenVINO GenAI 2024.5 Model: Gemma-7b-int4-ov - Device: CPU tokens/s > Higher Is Better a . 9.71 |===================================================================== b . 8.90 |=============================================================== c . 9.44 |=================================================================== OpenVINO GenAI 2024.5 Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPU tokens/s > Higher Is Better a . 31.43 |==================================================================== b . 30.36 |================================================================== c . 31.38 |==================================================================== OpenVINO GenAI 2024.5 Model: Falcon-7b-instruct-int4-ov - Device: CPU tokens/s > Higher Is Better a . 14.25 |==================================================================== b . 12.01 |========================================================= c . 14.06 |=================================================================== OpenVINO GenAI 2024.5 Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU tokens/s > Higher Is Better a . 19.83 |==================================================================== b . 18.58 |================================================================ c . 18.22 |============================================================== OpenVINO 2024.5 Model: Face Detection FP16 - Device: CPU ms < Lower Is Better a . 865.46 |================================================================== b . 867.81 |================================================================== c . 882.69 |=================================================================== OpenVINO 2024.5 Model: Person Detection FP16 - Device: CPU ms < Lower Is Better a . 90.96 |=================================================================== b . 91.66 |==================================================================== c . 91.40 |==================================================================== OpenVINO 2024.5 Model: Person Detection FP32 - Device: CPU ms < Lower Is Better a . 90.68 |=================================================================== b . 91.14 |=================================================================== c . 92.28 |==================================================================== OpenVINO 2024.5 Model: Vehicle Detection FP16 - Device: CPU ms < Lower Is Better a . 14.09 |==================================================================== b . 12.70 |============================================================= c . 12.78 |============================================================== OpenVINO 2024.5 Model: Face Detection FP16-INT8 - Device: CPU ms < Lower Is Better a . 467.21 |=============================================================== b . 449.00 |============================================================ c . 497.60 |=================================================================== OpenVINO 2024.5 Model: Face Detection Retail FP16 - Device: CPU ms < Lower Is Better a . 4.19 |==================================================================== b . 3.85 |=============================================================== c . 4.23 |===================================================================== OpenVINO 2024.5 Model: Road Segmentation ADAS FP16 - Device: CPU ms < Lower Is Better a . 31.72 |==================================================================== b . 29.22 |=============================================================== c . 29.99 |================================================================ OpenVINO 2024.5 Model: Vehicle Detection FP16-INT8 - Device: CPU ms < Lower Is Better a . 8.54 |===================================================================== b . 7.87 |================================================================ c . 8.04 |================================================================= OpenVINO 2024.5 Model: Weld Porosity Detection FP16 - Device: CPU ms < Lower Is Better a . 22.37 |==================================================================== b . 20.71 |=============================================================== c . 21.76 |================================================================== OpenVINO 2024.5 Model: Face Detection Retail FP16-INT8 - Device: CPU ms < Lower Is Better a . 6.27 |===================================================================== b . 5.82 |================================================================ c . 5.94 |================================================================= OpenVINO 2024.5 Model: Road Segmentation ADAS FP16-INT8 - Device: CPU ms < Lower Is Better a . 23.49 |==================================================================== b . 22.72 |================================================================== c . 21.89 |=============================================================== OpenVINO 2024.5 Model: Machine Translation EN To DE FP16 - Device: CPU ms < Lower Is Better a . 81.62 |==================================================================== b . 73.35 |============================================================= c . 74.77 |============================================================== OpenVINO 2024.5 Model: Weld Porosity Detection FP16-INT8 - Device: CPU ms < Lower Is Better a . 11.25 |==================================================================== b . 10.46 |=============================================================== c . 10.83 |================================================================= OpenVINO 2024.5 Model: Person Vehicle Bike Detection FP16 - Device: CPU ms < Lower Is Better a . 9.13 |===================================================================== b . 8.28 |=============================================================== c . 8.40 |=============================================================== OpenVINO 2024.5 Model: Noise Suppression Poconet-Like FP16 - Device: CPU ms < Lower Is Better a . 16.30 |==================================================================== b . 14.88 |============================================================== c . 14.96 |============================================================== OpenVINO 2024.5 Model: Handwritten English Recognition FP16 - Device: CPU ms < Lower Is Better a . 40.66 |==================================================================== b . 37.40 |=============================================================== c . 39.51 |================================================================== OpenVINO 2024.5 Model: Person Re-Identification Retail FP16 - Device: CPU ms < Lower Is Better a . 7.37 |===================================================================== b . 7.15 |=================================================================== c . 7.11 |=================================================================== OpenVINO 2024.5 Model: Age Gender Recognition Retail 0013 FP16 - Device: CPU ms < Lower Is Better a . 0.77 |===================================================================== b . 0.75 |=================================================================== c . 0.74 |================================================================== OpenVINO 2024.5 Model: Handwritten English Recognition FP16-INT8 - Device: CPU ms < Lower Is Better a . 39.27 |==================================================================== b . 35.29 |============================================================= c . 35.90 |============================================================== OpenVINO 2024.5 Model: Age Gender Recognition Retail 0013 FP16-INT8 - Device: CPU ms < Lower Is Better a . 0.53 |===================================================================== b . 0.49 |================================================================ c . 0.50 |================================================================= OpenVINO GenAI 2024.5 Model: Gemma-7b-int4-ov - Device: CPU - Time To First Token ms < Lower Is Better a . 204.63 |================================================================ b . 215.32 |=================================================================== c . 204.71 |================================================================ OpenVINO GenAI 2024.5 Model: Gemma-7b-int4-ov - Device: CPU - Time Per Output Token ms < Lower Is Better a . 102.99 |============================================================= b . 112.38 |=================================================================== c . 105.93 |=============================================================== OpenVINO GenAI 2024.5 Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPU - Time To First Token ms < Lower Is Better a . 34.26 |================================================================= b . 35.97 |==================================================================== c . 34.11 |================================================================ OpenVINO GenAI 2024.5 Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPU - Time Per Output Token ms < Lower Is Better a . 31.81 |================================================================== b . 32.93 |==================================================================== c . 31.87 |================================================================== OpenVINO GenAI 2024.5 Model: Falcon-7b-instruct-int4-ov - Device: CPU - Time To First Token ms < Lower Is Better a . 146.00 |====================================================== b . 181.63 |=================================================================== c . 148.94 |======================================================= OpenVINO GenAI 2024.5 Model: Falcon-7b-instruct-int4-ov - Device: CPU - Time Per Output Token ms < Lower Is Better a . 70.17 |========================================================= b . 83.28 |==================================================================== c . 71.15 |========================================================== OpenVINO GenAI 2024.5 Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU - Time To First Token ms < Lower Is Better a . 93.70 |============================================================= b . 100.02 |================================================================= c . 103.28 |=================================================================== OpenVINO GenAI 2024.5 Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU - Time Per Output Token ms < Lower Is Better a . 50.42 |============================================================== b . 53.82 |=================================================================== c . 54.90 |====================================================================