openvino nvidia grace ARMv8 Neoverse-V2 testing with a Pegatron JIMBO P4352 (00022432 BIOS) and ASPEED on Ubuntu 24.04 via the Phoronix Test Suite. a: Processor: ARMv8 Neoverse-V2 @ 3.47GHz (72 Cores), Motherboard: Pegatron JIMBO P4352 (00022432 BIOS), Memory: 1 x 480GB LPDDR5-6400MT/s NVIDIA 699-2G530-0236-RC1, Disk: 1000GB CT1000T700SSD3, Graphics: ASPEED, Network: 2 x Intel X550 OS: Ubuntu 24.04, Kernel: 6.8.0-49-generic-64k (aarch64), Compiler: GCC 13.2.0 + Clang 18.1.3 + CUDA 11.8, File-System: ext4, Screen Resolution: 1920x1200 b: Processor: ARMv8 Neoverse-V2 @ 3.47GHz (72 Cores), Motherboard: Pegatron JIMBO P4352 (00022432 BIOS), Memory: 1 x 480GB LPDDR5-6400MT/s NVIDIA 699-2G530-0236-RC1, Disk: 1000GB CT1000T700SSD3, Graphics: ASPEED, Network: 2 x Intel X550 OS: Ubuntu 24.04, Kernel: 6.8.0-49-generic-64k (aarch64), Compiler: GCC 13.2.0 + Clang 18.1.3 + CUDA 11.8, File-System: ext4, Screen Resolution: 1920x1200 c: Processor: ARMv8 Neoverse-V2 @ 3.47GHz (72 Cores), Motherboard: Pegatron JIMBO P4352 (00022432 BIOS), Memory: 1 x 480GB LPDDR5-6400MT/s NVIDIA 699-2G530-0236-RC1, Disk: 1000GB CT1000T700SSD3, Graphics: ASPEED, Network: 2 x Intel X550 OS: Ubuntu 24.04, Kernel: 6.8.0-49-generic-64k (aarch64), Compiler: GCC 13.2.0 + Clang 18.1.3 + CUDA 11.8, File-System: ext4, Screen Resolution: 1920x1200 d: Processor: ARMv8 Neoverse-V2 @ 3.47GHz (72 Cores), Motherboard: Pegatron JIMBO P4352 (00022432 BIOS), Memory: 1 x 480GB LPDDR5-6400MT/s NVIDIA 699-2G530-0236-RC1, Disk: 1000GB CT1000T700SSD3, Graphics: ASPEED, Network: 2 x Intel X550 OS: Ubuntu 24.04, Kernel: 6.8.0-49-generic-64k (aarch64), Compiler: GCC 13.2.0 + Clang 18.1.3 + CUDA 11.8, File-System: ext4, Screen Resolution: 1920x1200 OpenVINO 2024.5 Model: Face Detection FP16 - Device: CPU FPS > Higher Is Better a . 1.52 |==================================================================== b . 1.52 |==================================================================== c . 1.55 |===================================================================== d . 1.53 |==================================================================== OpenVINO 2024.5 Model: Face Detection FP16 - Device: CPU ms < Lower Is Better a . 11430.97 |================================================================= b . 11436.50 |================================================================= c . 11238.09 |================================================================ d . 11360.46 |================================================================= OpenVINO 2024.5 Model: Person Detection FP16 - Device: CPU FPS > Higher Is Better a . 7.41 |==================================================================== b . 7.34 |==================================================================== c . 7.48 |===================================================================== d . 7.40 |==================================================================== OpenVINO 2024.5 Model: Person Detection FP16 - Device: CPU ms < Lower Is Better a . 2411.48 |================================================================= b . 2437.90 |================================================================== c . 2385.09 |================================================================= d . 2410.59 |================================================================= OpenVINO 2024.5 Model: Person Detection FP32 - Device: CPU FPS > Higher Is Better a . 7.45 |===================================================================== b . 7.50 |===================================================================== c . 7.46 |===================================================================== d . 7.41 |==================================================================== OpenVINO 2024.5 Model: Person Detection FP32 - Device: CPU ms < Lower Is Better a . 2395.80 |================================================================== b . 2378.33 |================================================================= c . 2391.02 |================================================================== d . 2406.82 |================================================================== OpenVINO 2024.5 Model: Vehicle Detection FP16 - Device: CPU FPS > Higher Is Better a . 56.57 |================================================================== b . 51.71 |============================================================ c . 58.55 |==================================================================== d . 51.73 |============================================================ OpenVINO 2024.5 Model: Vehicle Detection FP16 - Device: CPU ms < Lower Is Better a . 319.46 |============================================================== b . 347.47 |=================================================================== c . 306.89 |=========================================================== d . 347.40 |=================================================================== OpenVINO 2024.5 Model: Face Detection FP16-INT8 - Device: CPU FPS > Higher Is Better a . 1.43 |===================================================================== b . 1.44 |===================================================================== c . 1.44 |===================================================================== d . 1.43 |===================================================================== OpenVINO 2024.5 Model: Face Detection FP16-INT8 - Device: CPU ms < Lower Is Better a . 12201.72 |================================================================= b . 12086.33 |================================================================ c . 12178.49 |================================================================= d . 12191.22 |================================================================= OpenVINO 2024.5 Model: Face Detection Retail FP16 - Device: CPU FPS > Higher Is Better a . 148.41 |=================================================================== b . 147.65 |=================================================================== c . 148.02 |=================================================================== d . 148.66 |=================================================================== OpenVINO 2024.5 Model: Face Detection Retail FP16 - Device: CPU ms < Lower Is Better a . 121.10 |=================================================================== b . 121.73 |=================================================================== c . 121.41 |=================================================================== d . 120.88 |=================================================================== OpenVINO 2024.5 Model: Road Segmentation ADAS FP16 - Device: CPU FPS > Higher Is Better a . 23.67 |==================================================================== b . 23.46 |=================================================================== c . 23.81 |==================================================================== d . 23.18 |================================================================== OpenVINO 2024.5 Model: Road Segmentation ADAS FP16 - Device: CPU ms < Lower Is Better a . 758.80 |================================================================== b . 765.14 |================================================================== c . 753.93 |================================================================= d . 774.08 |=================================================================== OpenVINO 2024.5 Model: Vehicle Detection FP16-INT8 - Device: CPU FPS > Higher Is Better a . 48.08 |==================================================================== b . 48.33 |==================================================================== c . 47.52 |=================================================================== d . 47.81 |=================================================================== OpenVINO 2024.5 Model: Vehicle Detection FP16-INT8 - Device: CPU ms < Lower Is Better a . 373.22 |================================================================== b . 371.04 |================================================================== c . 377.96 |=================================================================== d . 375.44 |=================================================================== OpenVINO 2024.5 Model: Weld Porosity Detection FP16 - Device: CPU FPS > Higher Is Better a . 73.61 |==================================================================== b . 73.30 |==================================================================== c . 72.75 |=================================================================== d . 73.48 |==================================================================== OpenVINO 2024.5 Model: Weld Porosity Detection FP16 - Device: CPU ms < Lower Is Better a . 243.99 |================================================================== b . 244.99 |================================================================== c . 246.92 |=================================================================== d . 244.38 |================================================================== OpenVINO 2024.5 Model: Face Detection Retail FP16-INT8 - Device: CPU FPS > Higher Is Better a . 243.74 |================================================================== b . 247.34 |=================================================================== c . 234.69 |================================================================ d . 245.24 |================================================================== OpenVINO 2024.5 Model: Face Detection Retail FP16-INT8 - Device: CPU ms < Lower Is Better a . 73.79 |================================================================= b . 72.71 |================================================================= c . 76.62 |==================================================================== d . 73.31 |================================================================= OpenVINO 2024.5 Model: Road Segmentation ADAS FP16-INT8 - Device: CPU FPS > Higher Is Better a . 12.81 |==================================================================== b . 12.81 |==================================================================== c . 12.77 |==================================================================== d . 12.83 |==================================================================== OpenVINO 2024.5 Model: Road Segmentation ADAS FP16-INT8 - Device: CPU ms < Lower Is Better a . 1390.97 |================================================================== b . 1396.38 |================================================================== c . 1394.95 |================================================================== d . 1385.22 |================================================================= OpenVINO 2024.5 Model: Machine Translation EN To DE FP16 - Device: CPU FPS > Higher Is Better a . 291.60 |=================================================================== b . 216.28 |================================================== c . 290.57 |=================================================================== d . 273.77 |=============================================================== OpenVINO 2024.5 Model: Machine Translation EN To DE FP16 - Device: CPU ms < Lower Is Better a . 61.68 |================================================== b . 83.15 |==================================================================== c . 61.84 |=================================================== d . 65.67 |====================================================== OpenVINO 2024.5 Model: Weld Porosity Detection FP16-INT8 - Device: CPU FPS > Higher Is Better a . 74.32 |================================================================== b . 74.42 |================================================================== c . 74.63 |=================================================================== d . 76.19 |==================================================================== OpenVINO 2024.5 Model: Weld Porosity Detection FP16-INT8 - Device: CPU ms < Lower Is Better a . 241.68 |=================================================================== b . 241.43 |=================================================================== c . 240.74 |=================================================================== d . 235.84 |================================================================= OpenVINO 2024.5 Model: Person Vehicle Bike Detection FP16 - Device: CPU FPS > Higher Is Better a . 123.10 |=================================================================== b . 122.96 |=================================================================== c . 122.93 |=================================================================== d . 123.43 |=================================================================== OpenVINO 2024.5 Model: Person Vehicle Bike Detection FP16 - Device: CPU ms < Lower Is Better a . 146.01 |=================================================================== b . 146.14 |=================================================================== c . 146.21 |=================================================================== d . 145.61 |=================================================================== OpenVINO 2024.5 Model: Noise Suppression Poconet-Like FP16 - Device: CPU FPS > Higher Is Better a . 260.47 |=================================================================== b . 252.67 |================================================================= c . 255.62 |================================================================== d . 255.69 |================================================================== OpenVINO 2024.5 Model: Noise Suppression Poconet-Like FP16 - Device: CPU ms < Lower Is Better a . 69.04 |================================================================== b . 71.18 |==================================================================== c . 70.37 |=================================================================== d . 70.34 |=================================================================== OpenVINO 2024.5 Model: Handwritten English Recognition FP16 - Device: CPU FPS > Higher Is Better a . 59.29 |=============================================================== b . 58.09 |============================================================== c . 57.50 |============================================================= d . 64.04 |==================================================================== OpenVINO 2024.5 Model: Handwritten English Recognition FP16 - Device: CPU ms < Lower Is Better a . 302.94 |================================================================= b . 308.96 |================================================================== c . 312.43 |=================================================================== d . 280.21 |============================================================ OpenVINO 2024.5 Model: Person Re-Identification Retail FP16 - Device: CPU FPS > Higher Is Better a . 81.55 |=================================================================== b . 81.93 |=================================================================== c . 81.75 |=================================================================== d . 82.57 |==================================================================== OpenVINO 2024.5 Model: Person Re-Identification Retail FP16 - Device: CPU ms < Lower Is Better a . 220.28 |=================================================================== b . 219.24 |=================================================================== c . 219.68 |=================================================================== d . 217.55 |================================================================== OpenVINO 2024.5 Model: Age Gender Recognition Retail 0013 FP16 - Device: CPU FPS > Higher Is Better a . 365.73 |=================================================================== b . 365.70 |=================================================================== c . 367.78 |=================================================================== d . 366.51 |=================================================================== OpenVINO 2024.5 Model: Age Gender Recognition Retail 0013 FP16 - Device: CPU ms < Lower Is Better a . 49.14 |==================================================================== b . 49.15 |==================================================================== c . 48.86 |==================================================================== d . 49.03 |==================================================================== OpenVINO 2024.5 Model: Handwritten English Recognition FP16-INT8 - Device: CPU FPS > Higher Is Better a . 62.80 |=================================================================== b . 62.64 |=================================================================== c . 62.93 |=================================================================== d . 63.41 |==================================================================== OpenVINO 2024.5 Model: Handwritten English Recognition FP16-INT8 - Device: CPU ms < Lower Is Better a . 286.07 |=================================================================== b . 286.88 |=================================================================== c . 285.46 |=================================================================== d . 283.00 |================================================================== OpenVINO 2024.5 Model: Age Gender Recognition Retail 0013 FP16-INT8 - Device: CPU FPS > Higher Is Better a . 368.63 |================================================================== b . 366.36 |================================================================== c . 372.99 |=================================================================== d . 367.55 |================================================================== OpenVINO 2024.5 Model: Age Gender Recognition Retail 0013 FP16-INT8 - Device: CPU ms < Lower Is Better a . 48.75 |==================================================================== b . 49.06 |==================================================================== c . 48.18 |=================================================================== d . 48.90 |==================================================================== OpenVINO GenAI 2024.5 Model: Gemma-7b-int4-ov - Device: CPU tokens/s > Higher Is Better a . 12.09 |==================================================================== b . 11.93 |=================================================================== c . 11.96 |=================================================================== d . 11.95 |=================================================================== OpenVINO GenAI 2024.5 Model: Gemma-7b-int4-ov - Device: CPU - Time To First Token ms < Lower Is Better a . 95.03 |=================================================================== b . 95.95 |==================================================================== c . 95.84 |==================================================================== d . 95.68 |==================================================================== OpenVINO GenAI 2024.5 Model: Gemma-7b-int4-ov - Device: CPU - Time Per Output Token ms < Lower Is Better a . 82.76 |=================================================================== b . 83.83 |==================================================================== c . 83.63 |==================================================================== d . 83.71 |==================================================================== OpenVINO GenAI 2024.5 Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPU tokens/s > Higher Is Better a . 54.99 |==================================================================== b . 54.93 |==================================================================== c . 54.16 |=================================================================== d . 53.22 |================================================================== OpenVINO GenAI 2024.5 Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPU - Time To First Token ms < Lower Is Better a . 27.90 |=================================================================== b . 28.03 |==================================================================== c . 28.17 |==================================================================== d . 27.96 |=================================================================== OpenVINO GenAI 2024.5 Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPU - Time Per Output Token ms < Lower Is Better a . 18.19 |================================================================== b . 18.20 |================================================================== c . 18.46 |=================================================================== d . 18.79 |==================================================================== OpenVINO GenAI 2024.5 Model: Falcon-7b-instruct-int4-ov - Device: CPU tokens/s > Higher Is Better a . 16.59 |==================================================================== b . 16.57 |==================================================================== c . 16.52 |==================================================================== d . 16.48 |==================================================================== OpenVINO GenAI 2024.5 Model: Falcon-7b-instruct-int4-ov - Device: CPU - Time To First Token ms < Lower Is Better a . 71.88 |==================================================================== b . 72.17 |==================================================================== c . 72.40 |==================================================================== d . 71.89 |==================================================================== OpenVINO GenAI 2024.5 Model: Falcon-7b-instruct-int4-ov - Device: CPU - Time Per Output Token ms < Lower Is Better a . 60.29 |==================================================================== b . 60.36 |==================================================================== c . 60.55 |==================================================================== d . 60.67 |==================================================================== OpenVINO GenAI 2024.5 Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU tokens/s > Higher Is Better a . 21.55 |==================================================================== b . 21.46 |==================================================================== c . 21.55 |==================================================================== d . 21.57 |==================================================================== OpenVINO GenAI 2024.5 Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU - Time To First Token ms < Lower Is Better a . 59.81 |==================================================================== b . 59.51 |==================================================================== c . 59.59 |==================================================================== d . 59.60 |==================================================================== OpenVINO GenAI 2024.5 Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU - Time Per Output Token ms < Lower Is Better a . 46.40 |==================================================================== b . 46.59 |==================================================================== c . 46.41 |==================================================================== d . 46.36 |====================================================================