ddddai AMD Ryzen 7 7840HS testing with a Framework Laptop 16 (AMD Ryzen 7040 ) FRANMZCP07 (03.01 BIOS) and AMD Radeon 780M 512MB on Ubuntu 24.04 via the Phoronix Test Suite. a: Processor: AMD Ryzen 7 7840HS @ 5.29GHz (8 Cores / 16 Threads), Motherboard: Framework Laptop 16 (AMD Ryzen 7040 ) FRANMZCP07 (03.01 BIOS), Chipset: AMD Device 14e8, Memory: 2 x 8GB DDR5-5600MT/s A-DATA AD5S56008G-B, Disk: 512GB Western Digital PC SN810 SDCPNRY-512G, Graphics: AMD Radeon 780M 512MB, Audio: AMD Navi 31 HDMI/DP, Network: MEDIATEK MT7922 802.11ax PCI OS: Ubuntu 24.04, Kernel: 6.8.0-48-generic (x86_64), Desktop: GNOME Shell 46.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.2~git2406200600.0ac0fb~oibaf~n (git-0ac0fbc 2024-06-20 noble-oibaf-ppa) (LLVM 17.0.6 DRM 3.57), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 2560x1600 b: Processor: AMD Ryzen 7 7840HS @ 5.29GHz (8 Cores / 16 Threads), Motherboard: Framework Laptop 16 (AMD Ryzen 7040 ) FRANMZCP07 (03.01 BIOS), Chipset: AMD Device 14e8, Memory: 2 x 8GB DDR5-5600MT/s A-DATA AD5S56008G-B, Disk: 512GB Western Digital PC SN810 SDCPNRY-512G, Graphics: AMD Radeon 780M 512MB, Audio: AMD Navi 31 HDMI/DP, Network: MEDIATEK MT7922 802.11ax PCI OS: Ubuntu 24.04, Kernel: 6.8.0-48-generic (x86_64), Desktop: GNOME Shell 46.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.2~git2406200600.0ac0fb~oibaf~n (git-0ac0fbc 2024-06-20 noble-oibaf-ppa) (LLVM 17.0.6 DRM 3.57), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 2560x1600 c: Processor: AMD Ryzen 7 7840HS @ 5.29GHz (8 Cores / 16 Threads), Motherboard: Framework Laptop 16 (AMD Ryzen 7040 ) FRANMZCP07 (03.01 BIOS), Chipset: AMD Device 14e8, Memory: 2 x 8GB DDR5-5600MT/s A-DATA AD5S56008G-B, Disk: 512GB Western Digital PC SN810 SDCPNRY-512G, Graphics: AMD Radeon 780M 512MB, Audio: AMD Navi 31 HDMI/DP, Network: MEDIATEK MT7922 802.11ax PCI OS: Ubuntu 24.04, Kernel: 6.8.0-48-generic (x86_64), Desktop: GNOME Shell 46.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.2~git2406200600.0ac0fb~oibaf~n (git-0ac0fbc 2024-06-20 noble-oibaf-ppa) (LLVM 17.0.6 DRM 3.57), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 2560x1600 d: Processor: AMD Ryzen 7 7840HS @ 5.29GHz (8 Cores / 16 Threads), Motherboard: Framework Laptop 16 (AMD Ryzen 7040 ) FRANMZCP07 (03.01 BIOS), Chipset: AMD Device 14e8, Memory: 2 x 8GB DDR5-5600MT/s A-DATA AD5S56008G-B, Disk: 512GB Western Digital PC SN810 SDCPNRY-512G, Graphics: AMD Radeon 780M 512MB, Audio: AMD Navi 31 HDMI/DP, Network: MEDIATEK MT7922 802.11ax PCI OS: Ubuntu 24.04, Kernel: 6.8.0-48-generic (x86_64), Desktop: GNOME Shell 46.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.2~git2406200600.0ac0fb~oibaf~n (git-0ac0fbc 2024-06-20 noble-oibaf-ppa) (LLVM 17.0.6 DRM 3.57), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 2560x1600 Llama.cpp b4154 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 41.38 |=================================================================== b . 41.10 |=================================================================== c . 41.61 |==================================================================== d . 41.71 |==================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 41.49 |==================================================================== b . 41.38 |=================================================================== c . 41.42 |=================================================================== d . 41.76 |==================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 44.22 |==================================================================== b . 43.45 |=================================================================== c . 43.56 |=================================================================== d . 43.77 |=================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 43.62 |=================================================================== b . 43.88 |=================================================================== c . 44.06 |==================================================================== d . 44.34 |==================================================================== OpenVINO GenAI 2024.5 Model: Gemma-7b-int4-ov - Device: CPU - Time Per Output Token ms < Lower Is Better a . 103.63 |=================================================================== b . 102.09 |================================================================== c . 102.37 |================================================================== d . 102.13 |================================================================== OpenVINO GenAI 2024.5 Model: Gemma-7b-int4-ov - Device: CPU - Time To First Token ms < Lower Is Better a . 188.15 |=================================================================== b . 182.66 |================================================================= c . 183.34 |================================================================= d . 183.33 |================================================================= OpenVINO GenAI 2024.5 Model: Gemma-7b-int4-ov - Device: CPU tokens/s > Higher Is Better a . 9.65 |==================================================================== b . 9.80 |===================================================================== c . 9.77 |===================================================================== d . 9.79 |===================================================================== OpenVINO GenAI 2024.5 Model: Falcon-7b-instruct-int4-ov - Device: CPU - Time Per Output Token ms < Lower Is Better a . 75.56 |==================================================================== b . 74.97 |=================================================================== c . 74.87 |=================================================================== d . 74.66 |=================================================================== OpenVINO GenAI 2024.5 Model: Falcon-7b-instruct-int4-ov - Device: CPU - Time To First Token ms < Lower Is Better a . 139.06 |=================================================================== b . 138.66 |=================================================================== c . 138.41 |=================================================================== d . 135.36 |================================================================= OpenVINO GenAI 2024.5 Model: Falcon-7b-instruct-int4-ov - Device: CPU tokens/s > Higher Is Better a . 13.23 |=================================================================== b . 13.34 |==================================================================== c . 13.36 |==================================================================== d . 13.39 |==================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 7.25 |===================================================================== b . 7.25 |===================================================================== c . 7.25 |===================================================================== d . 7.25 |===================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 7.64 |===================================================================== b . 7.63 |===================================================================== c . 7.63 |===================================================================== d . 7.63 |===================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 2048 Tokens Per Second > Higher Is Better a . 142.47 |=============================================================== b . 146.13 |================================================================= c . 146.86 |================================================================= d . 150.40 |=================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: Llama-3.1-Tulu-3-8B-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 44.83 |=================================================================== b . 45.25 |==================================================================== c . 44.76 |=================================================================== d . 45.09 |==================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: Mistral-7B-Instruct-v0.3-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 45.71 |==================================================================== b . 44.58 |================================================================== c . 45.05 |=================================================================== d . 45.20 |=================================================================== OpenVINO 2024.5 Model: Face Detection FP16 - Device: CPU ms < Lower Is Better a . 679.69 |=================================================================== b . 673.29 |================================================================== c . 677.43 |=================================================================== d . 678.48 |=================================================================== OpenVINO 2024.5 Model: Face Detection FP16 - Device: CPU FPS > Higher Is Better a . 5.87 |==================================================================== b . 5.93 |===================================================================== c . 5.88 |==================================================================== d . 5.89 |===================================================================== OpenVINO 2024.5 Model: Face Detection FP16-INT8 - Device: CPU ms < Lower Is Better a . 347.20 |=================================================================== b . 341.61 |================================================================== c . 339.00 |================================================================= d . 339.14 |================================================================= OpenVINO 2024.5 Model: Face Detection FP16-INT8 - Device: CPU FPS > Higher Is Better a . 11.50 |================================================================== b . 11.69 |=================================================================== c . 11.77 |==================================================================== d . 11.78 |==================================================================== OpenVINO 2024.5 Model: Machine Translation EN To DE FP16 - Device: CPU ms < Lower Is Better a . 70.89 |=================================================================== b . 71.09 |=================================================================== c . 71.68 |==================================================================== d . 70.85 |=================================================================== OpenVINO 2024.5 Model: Machine Translation EN To DE FP16 - Device: CPU FPS > Higher Is Better a . 56.36 |==================================================================== b . 56.23 |==================================================================== c . 55.77 |=================================================================== d . 56.39 |==================================================================== OpenVINO 2024.5 Model: Person Detection FP16 - Device: CPU ms < Lower Is Better a . 107.33 |=================================================================== b . 107.36 |=================================================================== c . 105.19 |================================================================== d . 106.71 |=================================================================== OpenVINO 2024.5 Model: Person Detection FP16 - Device: CPU FPS > Higher Is Better a . 37.22 |=================================================================== b . 37.23 |=================================================================== c . 37.98 |==================================================================== d . 37.45 |=================================================================== OpenVINO 2024.5 Model: Person Detection FP32 - Device: CPU ms < Lower Is Better a . 108.46 |=================================================================== b . 107.56 |================================================================== c . 108.21 |=================================================================== d . 107.74 |=================================================================== OpenVINO 2024.5 Model: Person Detection FP32 - Device: CPU FPS > Higher Is Better a . 36.86 |=================================================================== b . 37.15 |==================================================================== c . 36.94 |==================================================================== d . 37.08 |==================================================================== OpenVINO 2024.5 Model: Noise Suppression Poconet-Like FP16 - Device: CPU ms < Lower Is Better a . 14.59 |=================================================================== b . 14.78 |==================================================================== c . 14.45 |================================================================== d . 14.51 |=================================================================== OpenVINO 2024.5 Model: Noise Suppression Poconet-Like FP16 - Device: CPU FPS > Higher Is Better a . 543.71 |================================================================== b . 536.31 |================================================================= c . 549.16 |=================================================================== d . 546.60 |=================================================================== OpenVINO 2024.5 Model: Road Segmentation ADAS FP16-INT8 - Device: CPU ms < Lower Is Better a . 21.90 |==================================================================== b . 21.13 |================================================================== c . 21.05 |================================================================= d . 21.06 |================================================================= OpenVINO 2024.5 Model: Road Segmentation ADAS FP16-INT8 - Device: CPU FPS > Higher Is Better a . 182.16 |================================================================ b . 188.79 |=================================================================== c . 189.59 |=================================================================== d . 189.49 |=================================================================== OpenVINO 2024.5 Model: Person Vehicle Bike Detection FP16 - Device: CPU ms < Lower Is Better a . 7.23 |===================================================================== b . 7.18 |===================================================================== c . 7.23 |===================================================================== d . 7.14 |==================================================================== OpenVINO 2024.5 Model: Person Vehicle Bike Detection FP16 - Device: CPU FPS > Higher Is Better a . 550.99 |================================================================== b . 554.98 |=================================================================== c . 551.35 |================================================================== d . 558.37 |=================================================================== OpenVINO 2024.5 Model: Road Segmentation ADAS FP16 - Device: CPU ms < Lower Is Better a . 37.78 |================================================================== b . 36.67 |================================================================ c . 38.66 |==================================================================== d . 38.29 |=================================================================== OpenVINO 2024.5 Model: Road Segmentation ADAS FP16 - Device: CPU FPS > Higher Is Better a . 105.69 |================================================================= b . 108.90 |=================================================================== c . 103.29 |================================================================ d . 104.27 |================================================================ OpenVINO 2024.5 Model: Person Re-Identification Retail FP16 - Device: CPU ms < Lower Is Better a . 5.17 |===================================================================== b . 5.08 |==================================================================== c . 5.12 |==================================================================== d . 5.12 |==================================================================== OpenVINO 2024.5 Model: Person Re-Identification Retail FP16 - Device: CPU FPS > Higher Is Better a . 768.99 |================================================================== b . 783.60 |=================================================================== c . 777.34 |================================================================== d . 777.18 |================================================================== OpenVINO 2024.5 Model: Vehicle Detection FP16-INT8 - Device: CPU ms < Lower Is Better a . 6.74 |==================================================================== b . 6.76 |===================================================================== c . 6.79 |===================================================================== d . 6.74 |==================================================================== OpenVINO 2024.5 Model: Vehicle Detection FP16-INT8 - Device: CPU FPS > Higher Is Better a . 590.67 |=================================================================== b . 588.27 |=================================================================== c . 586.59 |=================================================================== d . 590.33 |=================================================================== OpenVINO 2024.5 Model: Face Detection Retail FP16-INT8 - Device: CPU ms < Lower Is Better a . 4.13 |==================================================================== b . 4.14 |===================================================================== c . 4.12 |==================================================================== d . 4.17 |===================================================================== OpenVINO 2024.5 Model: Face Detection Retail FP16-INT8 - Device: CPU FPS > Higher Is Better a . 1908.38 |================================================================== b . 1909.12 |================================================================== c . 1916.74 |================================================================== d . 1893.54 |================================================================= OpenVINO 2024.5 Model: Handwritten English Recognition FP16-INT8 - Device: CPU ms < Lower Is Better a . 23.79 |==================================================================== b . 23.58 |=================================================================== c . 23.79 |==================================================================== d . 23.84 |==================================================================== OpenVINO 2024.5 Model: Handwritten English Recognition FP16-INT8 - Device: CPU FPS > Higher Is Better a . 335.59 |================================================================== b . 338.64 |=================================================================== c . 335.54 |================================================================== d . 334.86 |================================================================== OpenVINO 2024.5 Model: Handwritten English Recognition FP16 - Device: CPU ms < Lower Is Better a . 27.28 |==================================================================== b . 27.45 |==================================================================== c . 26.58 |================================================================== d . 26.66 |================================================================== OpenVINO 2024.5 Model: Handwritten English Recognition FP16 - Device: CPU FPS > Higher Is Better a . 292.71 |================================================================= b . 290.66 |================================================================= c . 300.37 |=================================================================== d . 299.44 |=================================================================== OpenVINO 2024.5 Model: Vehicle Detection FP16 - Device: CPU ms < Lower Is Better a . 16.13 |=================================================================== b . 16.25 |==================================================================== c . 16.31 |==================================================================== d . 15.95 |================================================================== OpenVINO 2024.5 Model: Vehicle Detection FP16 - Device: CPU FPS > Higher Is Better a . 247.14 |================================================================== b . 245.32 |================================================================== c . 244.42 |================================================================== d . 250.01 |=================================================================== OpenVINO 2024.5 Model: Weld Porosity Detection FP16 - Device: CPU ms < Lower Is Better a . 13.47 |=================================================================== b . 13.62 |==================================================================== c . 13.58 |==================================================================== d . 13.54 |==================================================================== OpenVINO 2024.5 Model: Weld Porosity Detection FP16 - Device: CPU FPS > Higher Is Better a . 590.41 |=================================================================== b . 583.92 |================================================================== c . 586.20 |=================================================================== d . 587.52 |=================================================================== OpenVINO 2024.5 Model: Age Gender Recognition Retail 0013 FP16-INT8 - Device: CPU ms < Lower Is Better a . 0.35 |===================================================================== b . 0.35 |===================================================================== c . 0.35 |===================================================================== d . 0.35 |===================================================================== OpenVINO 2024.5 Model: Age Gender Recognition Retail 0013 FP16-INT8 - Device: CPU FPS > Higher Is Better a . 21199.70 |================================================================= b . 21264.05 |================================================================= c . 21235.24 |================================================================= d . 21253.78 |================================================================= OpenVINO 2024.5 Model: Weld Porosity Detection FP16-INT8 - Device: CPU ms < Lower Is Better a . 7.04 |===================================================================== b . 7.09 |===================================================================== c . 7.07 |===================================================================== d . 7.05 |===================================================================== OpenVINO 2024.5 Model: Weld Porosity Detection FP16-INT8 - Device: CPU FPS > Higher Is Better a . 1121.63 |================================================================== b . 1112.06 |================================================================= c . 1116.04 |================================================================== d . 1120.88 |================================================================== OpenVINO 2024.5 Model: Face Detection Retail FP16 - Device: CPU ms < Lower Is Better a . 2.96 |==================================================================== b . 2.98 |===================================================================== c . 2.98 |===================================================================== d . 2.99 |===================================================================== OpenVINO 2024.5 Model: Face Detection Retail FP16 - Device: CPU FPS > Higher Is Better a . 1343.56 |================================================================== b . 1332.62 |================================================================= c . 1330.69 |================================================================= d . 1327.47 |================================================================= OpenVINO 2024.5 Model: Age Gender Recognition Retail 0013 FP16 - Device: CPU ms < Lower Is Better a . 0.48 |==================================================================== b . 0.49 |===================================================================== c . 0.49 |===================================================================== d . 0.49 |===================================================================== OpenVINO 2024.5 Model: Age Gender Recognition Retail 0013 FP16 - Device: CPU FPS > Higher Is Better a . 15348.57 |================================================================= b . 15326.37 |================================================================= c . 15335.47 |================================================================= d . 15342.91 |================================================================= OpenVINO GenAI 2024.5 Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPU - Time Per Output Token ms < Lower Is Better a . 40.31 |==================================================================== b . 40.41 |==================================================================== c . 40.37 |==================================================================== d . 40.42 |==================================================================== OpenVINO GenAI 2024.5 Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPU - Time To First Token ms < Lower Is Better a . 43.01 |==================================================================== b . 43.20 |==================================================================== c . 43.16 |==================================================================== d . 43.12 |==================================================================== OpenVINO GenAI 2024.5 Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPU tokens/s > Higher Is Better a . 24.81 |==================================================================== b . 24.74 |==================================================================== c . 24.77 |==================================================================== d . 24.74 |==================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 1024 Tokens Per Second > Higher Is Better a . 155.44 |================================================================ b . 152.29 |=============================================================== c . 161.80 |=================================================================== d . 155.83 |================================================================= OpenVINO GenAI 2024.5 Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU - Time Per Output Token ms < Lower Is Better a . 52.53 |==================================================================== b . 52.38 |==================================================================== c . 52.23 |==================================================================== d . 52.58 |==================================================================== OpenVINO GenAI 2024.5 Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU - Time To First Token ms < Lower Is Better a . 86.75 |==================================================================== b . 86.94 |==================================================================== c . 86.87 |==================================================================== d . 86.74 |==================================================================== OpenVINO GenAI 2024.5 Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU tokens/s > Higher Is Better a . 19.03 |==================================================================== b . 19.09 |==================================================================== c . 19.15 |==================================================================== d . 19.02 |==================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Prompt Processing 512 Tokens Per Second > Higher Is Better a . 166.26 |=============================================================== b . 162.09 |============================================================= c . 166.80 |=============================================================== d . 178.14 |=================================================================== Llama.cpp b4154 Backend: CPU BLAS - Model: granite-3.0-3b-a800m-instruct-Q8_0 - Test: Text Generation 128 Tokens Per Second > Higher Is Better a . 54.63 |==================================================================== b . 54.64 |==================================================================== c . 54.46 |==================================================================== d . 54.69 |====================================================================