ai ryzen AMD Ryzen AI 9 HX 370 testing with a ASUS Zenbook S 16 UM5606WA_UM5606WA UM5606WA v1.0 (UM5606WA.308 BIOS) and AMD Radeon 512MB on Ubuntu 24.10 via the Phoronix Test Suite. a: Processor: AMD Ryzen AI 9 HX 370 @ 4.37GHz (12 Cores / 24 Threads), Motherboard: ASUS Zenbook S 16 UM5606WA_UM5606WA UM5606WA v1.0 (UM5606WA.308 BIOS), Chipset: AMD Device 1507, Memory: 4 x 8GB LPDDR5-7500MT/s Samsung K3KL9L90CM-MGCT, Disk: 1024GB MTFDKBA1T0QFM-1BD1AABGB, Graphics: AMD Radeon 512MB, Audio: AMD Rembrandt Radeon HD Audio, Network: MEDIATEK Device 7925 OS: Ubuntu 24.10, Kernel: 6.11.0-rc6-phx (x86_64), Desktop: GNOME Shell 47.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.2.3-1ubuntu1 (LLVM 19.1.0 DRM 3.58), Compiler: GCC 14.2.0, File-System: ext4, Screen Resolution: 2880x1800 b: Processor: AMD Ryzen AI 9 HX 370 @ 4.37GHz (12 Cores / 24 Threads), Motherboard: ASUS Zenbook S 16 UM5606WA_UM5606WA UM5606WA v1.0 (UM5606WA.308 BIOS), Chipset: AMD Device 1507, Memory: 4 x 8GB LPDDR5-7500MT/s Samsung K3KL9L90CM-MGCT, Disk: 1024GB MTFDKBA1T0QFM-1BD1AABGB, Graphics: AMD Radeon 512MB, Audio: AMD Rembrandt Radeon HD Audio, Network: MEDIATEK Device 7925 OS: Ubuntu 24.10, Kernel: 6.11.0-rc6-phx (x86_64), Desktop: GNOME Shell 47.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.2.3-1ubuntu1 (LLVM 19.1.0 DRM 3.58), Compiler: GCC 14.2.0, File-System: ext4, Screen Resolution: 2880x1800 c: Processor: AMD Ryzen AI 9 HX 370 @ 4.37GHz (12 Cores / 24 Threads), Motherboard: ASUS Zenbook S 16 UM5606WA_UM5606WA UM5606WA v1.0 (UM5606WA.308 BIOS), Chipset: AMD Device 1507, Memory: 4 x 8GB LPDDR5-7500MT/s Samsung K3KL9L90CM-MGCT, Disk: 1024GB MTFDKBA1T0QFM-1BD1AABGB, Graphics: AMD Radeon 512MB, Audio: AMD Rembrandt Radeon HD Audio, Network: MEDIATEK Device 7925 OS: Ubuntu 24.10, Kernel: 6.11.0-rc6-phx (x86_64), Desktop: GNOME Shell 47.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.2.3-1ubuntu1 (LLVM 19.1.0 DRM 3.58), Compiler: GCC 14.2.0, File-System: ext4, Screen Resolution: 2880x1800 d: Processor: AMD Ryzen AI 9 HX 370 @ 4.37GHz (12 Cores / 24 Threads), Motherboard: ASUS Zenbook S 16 UM5606WA_UM5606WA UM5606WA v1.0 (UM5606WA.308 BIOS), Chipset: AMD Device 1507, Memory: 4 x 8GB LPDDR5-7500MT/s Samsung K3KL9L90CM-MGCT, Disk: 1024GB MTFDKBA1T0QFM-1BD1AABGB, Graphics: AMD Radeon 512MB, Audio: AMD Rembrandt Radeon HD Audio, Network: MEDIATEK Device 7925 OS: Ubuntu 24.10, Kernel: 6.11.0-rc6-phx (x86_64), Desktop: GNOME Shell 47.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.2.3-1ubuntu1 (LLVM 19.1.0 DRM 3.58), Compiler: GCC 14.2.0, File-System: ext4, Screen Resolution: 2880x1800 e: Processor: AMD Ryzen AI 9 HX 370 @ 4.37GHz (12 Cores / 24 Threads), Motherboard: ASUS Zenbook S 16 UM5606WA_UM5606WA UM5606WA v1.0 (UM5606WA.308 BIOS), Chipset: AMD Device 1507, Memory: 4 x 8GB LPDDR5-7500MT/s Samsung K3KL9L90CM-MGCT, Disk: 1024GB MTFDKBA1T0QFM-1BD1AABGB, Graphics: AMD Radeon 512MB, Audio: AMD Rembrandt Radeon HD Audio, Network: MEDIATEK Device 7925 OS: Ubuntu 24.10, Kernel: 6.11.0-rc6-phx (x86_64), Desktop: GNOME Shell 47.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.2.3-1ubuntu1 (LLVM 19.1.0 DRM 3.58), Compiler: GCC 14.2.0, File-System: ext4, Screen Resolution: 2880x1800 OpenVINO 2024.5 Model: Face Detection FP16 - Device: CPU FPS > Higher Is Better a . 5.44 |===================================================================== b . 5.27 |=================================================================== c . 4.55 |========================================================== d . 5.38 |==================================================================== e . 4.38 |======================================================== OpenVINO 2024.5 Model: Person Detection FP16 - Device: CPU FPS > Higher Is Better a . 52.06 |==================================================================== b . 50.83 |================================================================== c . 45.61 |============================================================ d . 51.53 |=================================================================== e . 42.62 |======================================================== OpenVINO 2024.5 Model: Person Detection FP32 - Device: CPU FPS > Higher Is Better a . 51.79 |==================================================================== b . 44.21 |========================================================== c . 40.96 |====================================================== d . 51.35 |=================================================================== e . 40.18 |===================================================== OpenVINO 2024.5 Model: Vehicle Detection FP16 - Device: CPU FPS > Higher Is Better a . 365.47 |=================================================================== b . 266.56 |================================================= c . 268.07 |================================================= d . 363.73 |=================================================================== e . 272.03 |================================================== OpenVINO 2024.5 Model: Face Detection FP16-INT8 - Device: CPU FPS > Higher Is Better a . 10.53 |==================================================================== b . 7.60 |================================================= c . 8.12 |==================================================== d . 10.50 |==================================================================== e . 8.27 |===================================================== OpenVINO 2024.5 Model: Face Detection Retail FP16 - Device: CPU FPS > Higher Is Better a . 1169.64 |=============================================================== b . 870.25 |=============================================== c . 1010.95 |======================================================= d . 1215.99 |================================================================== e . 1025.50 |======================================================== OpenVINO 2024.5 Model: Road Segmentation ADAS FP16 - Device: CPU FPS > Higher Is Better a . 132.87 |============================================================== b . 128.34 |============================================================ c . 143.25 |=================================================================== d . 140.79 |================================================================== e . 138.77 |================================================================= OpenVINO 2024.5 Model: Vehicle Detection FP16-INT8 - Device: CPU FPS > Higher Is Better a . 444.31 |================================================================ b . 468.30 |=================================================================== c . 458.91 |================================================================== d . 429.24 |============================================================= e . 466.61 |=================================================================== OpenVINO 2024.5 Model: Weld Porosity Detection FP16 - Device: CPU FPS > Higher Is Better a . 422.25 |=================================================================== b . 404.47 |================================================================ c . 392.61 |============================================================== d . 401.70 |================================================================ e . 415.85 |================================================================== OpenVINO 2024.5 Model: Face Detection Retail FP16-INT8 - Device: CPU FPS > Higher Is Better a . 1640.62 |================================================================== b . 1490.61 |============================================================ c . 1471.58 |=========================================================== d . 1524.99 |============================================================= e . 1527.76 |============================================================= OpenVINO 2024.5 Model: Road Segmentation ADAS FP16-INT8 - Device: CPU FPS > Higher Is Better a . 206.96 |=================================================================== b . 183.64 |=========================================================== c . 193.78 |=============================================================== d . 197.59 |================================================================ e . 194.80 |=============================================================== OpenVINO 2024.5 Model: Machine Translation EN To DE FP16 - Device: CPU FPS > Higher Is Better a . 54.73 |==================================================================== b . 49.22 |============================================================= c . 52.50 |================================================================= d . 53.59 |=================================================================== e . 52.41 |================================================================= OpenVINO 2024.5 Model: Weld Porosity Detection FP16-INT8 - Device: CPU FPS > Higher Is Better a . 844.93 |=================================================================== b . 798.01 |=============================================================== c . 828.11 |================================================================== d . 814.77 |================================================================= e . 836.23 |================================================================== OpenVINO 2024.5 Model: Person Vehicle Bike Detection FP16 - Device: CPU FPS > Higher Is Better a . 497.68 |================================================================== b . 490.51 |================================================================= c . 492.29 |================================================================= d . 486.65 |================================================================ e . 508.37 |=================================================================== OpenVINO 2024.5 Model: Noise Suppression Poconet-Like FP16 - Device: CPU FPS > Higher Is Better a . 621.44 |================================================================== b . 599.61 |================================================================ c . 606.01 |================================================================ d . 590.51 |=============================================================== e . 631.37 |=================================================================== OpenVINO 2024.5 Model: Handwritten English Recognition FP16 - Device: CPU FPS > Higher Is Better a . 208.99 |=================================================================== b . 200.66 |================================================================ c . 190.24 |============================================================= d . 193.27 |============================================================== e . 201.09 |================================================================ OpenVINO 2024.5 Model: Person Re-Identification Retail FP16 - Device: CPU FPS > Higher Is Better a . 593.23 |=================================================================== b . 564.28 |================================================================ c . 561.60 |=============================================================== d . 548.16 |============================================================== e . 578.76 |================================================================= OpenVINO 2024.5 Model: Age Gender Recognition Retail 0013 FP16 - Device: CPU FPS > Higher Is Better a . 11718.11 |================================================================= b . 10945.61 |============================================================= c . 11406.75 |=============================================================== d . 11135.80 |============================================================== e . 11583.86 |================================================================ OpenVINO 2024.5 Model: Handwritten English Recognition FP16-INT8 - Device: CPU FPS > Higher Is Better a . 249.42 |================================================================ b . 238.70 |============================================================= c . 250.62 |================================================================ d . 248.20 |=============================================================== e . 262.58 |=================================================================== OpenVINO 2024.5 Model: Age Gender Recognition Retail 0013 FP16-INT8 - Device: CPU FPS > Higher Is Better a . 16399.53 |============================================================== b . 16390.57 |============================================================== c . 16445.83 |============================================================== d . 16126.97 |============================================================= e . 17219.19 |================================================================= Llama.cpp b4154 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 10.01 |==================================================================== b . 9.99 |==================================================================== c . 9.96 |=================================================================== d . 9.83 |=================================================================== e . 10.04 |==================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 33.32 |==================================================================== b . 32.83 |=================================================================== c . 32.71 |=================================================================== d . 31.39 |================================================================ e . 32.72 |=================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 31.81 |================================================================== b . 31.75 |================================================================== c . 31.59 |================================================================== d . 31.42 |================================================================= e . 32.67 |==================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 30.09 |==================================================================== b . 30.13 |==================================================================== c . 29.97 |=================================================================== d . 29.90 |=================================================================== e . 30.22 |==================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 10.32 |=================================================================== b . 10.29 |=================================================================== c . 10.41 |==================================================================== d . 10.28 |=================================================================== e . 10.43 |==================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 33.00 |=================================================================== b . 32.38 |================================================================== c . 32.72 |=================================================================== d . 32.06 |================================================================== e . 33.26 |==================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 32.05 |==================================================================== b . 31.98 |==================================================================== c . 31.52 |=================================================================== d . 31.19 |================================================================== e . 31.93 |==================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 30.05 |================================================================ b . 31.77 |==================================================================== c . 30.11 |================================================================ d . 29.51 |=============================================================== e . 29.76 |================================================================ Llama.cpp b4154 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 55.19 |==================================================================== b . 54.06 |=================================================================== c . 55.18 |==================================================================== d . 54.03 |=================================================================== e . 54.66 |=================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 138.56 |================================================================== b . 140.61 |=================================================================== c . 127.48 |============================================================= d . 126.06 |============================================================ e . 137.03 |================================================================= Llama.cpp b4154 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 154.30 |=================================================================== b . 140.01 |============================================================= c . 149.21 |================================================================= d . 124.44 |====================================================== e . 135.17 |=========================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 129.98 |============================================================= b . 132.33 |============================================================== c . 120.58 |========================================================= d . 113.93 |====================================================== e . 142.42 |=================================================================== OpenVINO GenAI 2024.5 Model: Gemma-7b-int4-ov - Device: CPU tokens/s > Higher Is Better a . 8.37 |===================================================================== b . 8.33 |===================================================================== c . 8.24 |==================================================================== d . 8.20 |==================================================================== e . 8.28 |==================================================================== OpenVINO GenAI 2024.5 Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPU tokens/s > Higher Is Better a . 28.44 |==================================================================== b . 28.43 |==================================================================== c . 28.48 |==================================================================== d . 28.15 |=================================================================== e . 28.03 |=================================================================== OpenVINO GenAI 2024.5 Model: Falcon-7b-instruct-int4-ov - Device: CPU tokens/s > Higher Is Better a . 11.36 |================================================================ b . 11.81 |=================================================================== c . 11.66 |================================================================== d . 11.72 |================================================================== e . 12.01 |==================================================================== OpenVINO GenAI 2024.5 Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU tokens/s > Higher Is Better a . 17.04 |==================================================================== b . 17.02 |=================================================================== c . 17.16 |==================================================================== d . 16.97 |=================================================================== e . 16.86 |=================================================================== OpenVINO 2024.5 Model: Face Detection FP16 - Device: CPU ms < Lower Is Better a . 1100.13 |===================================================== b . 1134.91 |======================================================= c . 1309.47 |=============================================================== d . 1106.04 |===================================================== e . 1365.09 |================================================================== OpenVINO 2024.5 Model: Person Detection FP16 - Device: CPU ms < Lower Is Better a . 115.12 |======================================================= b . 117.92 |======================================================== c . 131.38 |=============================================================== d . 116.34 |======================================================= e . 140.66 |=================================================================== OpenVINO 2024.5 Model: Person Detection FP32 - Device: CPU ms < Lower Is Better a . 115.72 |==================================================== b . 135.46 |============================================================= c . 146.29 |================================================================== d . 116.71 |==================================================== e . 149.10 |=================================================================== OpenVINO 2024.5 Model: Vehicle Detection FP16 - Device: CPU ms < Lower Is Better a . 16.35 |================================================== b . 22.45 |==================================================================== c . 22.32 |==================================================================== d . 16.42 |================================================== e . 22.00 |=================================================================== OpenVINO 2024.5 Model: Face Detection FP16-INT8 - Device: CPU ms < Lower Is Better a . 568.10 |================================================ b . 787.26 |=================================================================== c . 736.30 |=============================================================== d . 569.98 |================================================= e . 722.78 |============================================================== OpenVINO 2024.5 Model: Face Detection Retail FP16 - Device: CPU ms < Lower Is Better a . 5.06 |=================================================== b . 6.83 |===================================================================== c . 5.87 |=========================================================== d . 4.86 |================================================= e . 5.78 |========================================================== OpenVINO 2024.5 Model: Road Segmentation ADAS FP16 - Device: CPU ms < Lower Is Better a . 45.08 |================================================================== b . 46.67 |==================================================================== c . 41.81 |============================================================= d . 42.53 |============================================================== e . 43.16 |=============================================================== OpenVINO 2024.5 Model: Vehicle Detection FP16-INT8 - Device: CPU ms < Lower Is Better a . 13.45 |================================================================== b . 12.76 |============================================================== c . 13.02 |================================================================ d . 13.92 |==================================================================== e . 12.80 |=============================================================== OpenVINO 2024.5 Model: Weld Porosity Detection FP16 - Device: CPU ms < Lower Is Better a . 28.29 |=============================================================== b . 29.53 |================================================================== c . 30.41 |==================================================================== d . 29.73 |================================================================== e . 28.71 |================================================================ OpenVINO 2024.5 Model: Face Detection Retail FP16-INT8 - Device: CPU ms < Lower Is Better a . 7.24 |============================================================== b . 7.97 |==================================================================== c . 8.08 |===================================================================== d . 7.79 |=================================================================== e . 7.78 |================================================================== OpenVINO 2024.5 Model: Road Segmentation ADAS FP16-INT8 - Device: CPU ms < Lower Is Better a . 28.94 |============================================================ b . 32.61 |==================================================================== c . 30.91 |================================================================ d . 30.31 |=============================================================== e . 30.75 |================================================================ OpenVINO 2024.5 Model: Machine Translation EN To DE FP16 - Device: CPU ms < Lower Is Better a . 109.56 |============================================================ b . 121.82 |=================================================================== c . 114.18 |=============================================================== d . 111.84 |============================================================== e . 114.35 |=============================================================== OpenVINO 2024.5 Model: Weld Porosity Detection FP16-INT8 - Device: CPU ms < Lower Is Better a . 14.10 |================================================================ b . 14.94 |==================================================================== c . 14.39 |================================================================= d . 14.63 |=================================================================== e . 14.25 |================================================================= OpenVINO 2024.5 Model: Person Vehicle Bike Detection FP16 - Device: CPU ms < Lower Is Better a . 12.00 |================================================================== b . 12.18 |=================================================================== c . 12.14 |=================================================================== d . 12.28 |==================================================================== e . 11.76 |================================================================= OpenVINO 2024.5 Model: Noise Suppression Poconet-Like FP16 - Device: CPU ms < Lower Is Better a . 19.11 |================================================================= b . 19.82 |=================================================================== c . 19.61 |================================================================== d . 20.13 |==================================================================== e . 18.81 |================================================================ OpenVINO 2024.5 Model: Handwritten English Recognition FP16 - Device: CPU ms < Lower Is Better a . 57.28 |============================================================== b . 59.65 |================================================================ c . 62.92 |==================================================================== d . 61.93 |=================================================================== e . 59.53 |================================================================ OpenVINO 2024.5 Model: Person Re-Identification Retail FP16 - Device: CPU ms < Lower Is Better a . 10.06 |=============================================================== b . 10.58 |================================================================== c . 10.62 |================================================================== d . 10.89 |==================================================================== e . 10.31 |================================================================ OpenVINO 2024.5 Model: Age Gender Recognition Retail 0013 FP16 - Device: CPU ms < Lower Is Better a . 0.94 |================================================================ b . 1.01 |===================================================================== c . 0.97 |================================================================== d . 0.99 |==================================================================== e . 0.95 |================================================================= OpenVINO 2024.5 Model: Handwritten English Recognition FP16-INT8 - Device: CPU ms < Lower Is Better a . 47.98 |================================================================= b . 50.13 |==================================================================== c . 47.74 |================================================================= d . 48.22 |================================================================= e . 45.56 |============================================================== OpenVINO 2024.5 Model: Age Gender Recognition Retail 0013 FP16-INT8 - Device: CPU ms < Lower Is Better a . 0.66 |==================================================================== b . 0.66 |==================================================================== c . 0.66 |==================================================================== d . 0.67 |===================================================================== e . 0.63 |================================================================= OpenVINO GenAI 2024.5 Model: Gemma-7b-int4-ov - Device: CPU - Time To First Token ms < Lower Is Better a . 231.13 |================================================================ b . 235.33 |================================================================= c . 237.72 |================================================================== d . 242.38 |=================================================================== e . 239.26 |================================================================== OpenVINO GenAI 2024.5 Model: Gemma-7b-int4-ov - Device: CPU - Time Per Output Token ms < Lower Is Better a . 119.42 |================================================================== b . 120.03 |================================================================== c . 121.37 |=================================================================== d . 121.97 |=================================================================== e . 120.84 |================================================================== OpenVINO GenAI 2024.5 Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPU - Time To First Token ms < Lower Is Better a . 39.22 |==================================================================== b . 39.10 |=================================================================== c . 38.84 |=================================================================== d . 39.20 |=================================================================== e . 39.50 |==================================================================== OpenVINO GenAI 2024.5 Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPU - Time Per Output Token ms < Lower Is Better a . 35.16 |=================================================================== b . 35.18 |=================================================================== c . 35.11 |=================================================================== d . 35.52 |==================================================================== e . 35.67 |==================================================================== OpenVINO GenAI 2024.5 Model: Falcon-7b-instruct-int4-ov - Device: CPU - Time To First Token ms < Lower Is Better a . 182.34 |=================================================================== b . 177.40 |================================================================= c . 178.52 |================================================================== d . 181.54 |=================================================================== e . 180.07 |================================================================== OpenVINO GenAI 2024.5 Model: Falcon-7b-instruct-int4-ov - Device: CPU - Time Per Output Token ms < Lower Is Better a . 88.04 |==================================================================== b . 84.70 |================================================================= c . 85.80 |================================================================== d . 85.32 |================================================================== e . 83.23 |================================================================ OpenVINO GenAI 2024.5 Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU - Time To First Token ms < Lower Is Better a . 109.71 |================================================================== b . 109.04 |================================================================== c . 110.78 |=================================================================== d . 110.53 |================================================================== e . 111.40 |=================================================================== OpenVINO GenAI 2024.5 Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU - Time Per Output Token ms < Lower Is Better a . 58.68 |=================================================================== b . 58.77 |=================================================================== c . 58.28 |=================================================================== d . 58.92 |==================================================================== e . 59.30 |====================================================================