altra openvino
ARMv8 Neoverse-N1 testing with a System76 Thelio Astra (3.02 BIOS) and NVIDIA RTX A400/PCIe 4GB on Ubuntu 24.04 via the Phoronix Test Suite.


ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB -: 

	Processor: ARMv8 Neoverse-N1 @ 3.00GHz (128 Cores), Motherboard: System76 Thelio Astra (3.02 BIOS), Chipset: Ampere Computing LLC Altra PCI Root Complex A, Memory: 8 x 32GB DDR4-3200MT/s Micron 18ASF4G72PDZ-3G2F1, Disk: 1024GB KINGSTON SKC3000S1024G, Graphics: NVIDIA RTX A400/PCIe 4GB, Audio: NVIDIA Device 2291, Monitor: DELL P2415Q, Network: 2 x Intel X550 + Intel I210

	OS: Ubuntu 24.04, Kernel: 6.8.0-48-generic-64k (aarch64), Desktop: GNOME Shell 46.0, Display Server: X Server, Display Driver: NVIDIA 550.120, OpenGL: 4.6.0, Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 3840x2160


OpenVINO GenAI 2024.5
Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU
tokens/s > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 10.10 |=======================


OpenVINO GenAI 2024.5
Model: Falcon-7b-instruct-int4-ov - Device: CPU
tokens/s > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 9.85 |========================


OpenVINO GenAI 2024.5
Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPU
tokens/s > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 19.76 |=======================


OpenVINO GenAI 2024.5
Model: Gemma-7b-int4-ov - Device: CPU
tokens/s > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 6.00 |========================


OpenVINO 2024.5
Model: Age Gender Recognition Retail 0013 FP16-INT8 - Device: CPU
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 17.01 |=======================


OpenVINO 2024.5
Model: Age Gender Recognition Retail 0013 FP16-INT8 - Device: CPU
FPS > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 1880.88 |=====================


OpenVINO 2024.5
Model: Handwritten English Recognition FP16-INT8 - Device: CPU
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 214.64 |======================


OpenVINO 2024.5
Model: Handwritten English Recognition FP16-INT8 - Device: CPU
FPS > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 148.84 |======================


OpenVINO 2024.5
Model: Age Gender Recognition Retail 0013 FP16 - Device: CPU
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 17.55 |=======================


OpenVINO 2024.5
Model: Age Gender Recognition Retail 0013 FP16 - Device: CPU
FPS > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 1822.54 |=====================


OpenVINO 2024.5
Model: Person Re-Identification Retail FP16 - Device: CPU
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 178.76 |======================


OpenVINO 2024.5
Model: Person Re-Identification Retail FP16 - Device: CPU
FPS > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 178.91 |======================


OpenVINO 2024.5
Model: Handwritten English Recognition FP16 - Device: CPU
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 184.02 |======================


OpenVINO 2024.5
Model: Handwritten English Recognition FP16 - Device: CPU
FPS > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 173.59 |======================


OpenVINO 2024.5
Model: Noise Suppression Poconet-Like FP16 - Device: CPU
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 352.06 |======================


OpenVINO 2024.5
Model: Noise Suppression Poconet-Like FP16 - Device: CPU
FPS > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 90.76 |=======================


OpenVINO 2024.5
Model: Person Vehicle Bike Detection FP16 - Device: CPU
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 108.51 |======================


OpenVINO 2024.5
Model: Person Vehicle Bike Detection FP16 - Device: CPU
FPS > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 294.75 |======================


OpenVINO 2024.5
Model: Weld Porosity Detection FP16-INT8 - Device: CPU
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 61.36 |=======================


OpenVINO 2024.5
Model: Weld Porosity Detection FP16-INT8 - Device: CPU
FPS > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 521.12 |======================


OpenVINO 2024.5
Model: Machine Translation EN To DE FP16 - Device: CPU
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 106.44 |======================


OpenVINO 2024.5
Model: Machine Translation EN To DE FP16 - Device: CPU
FPS > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 300.36 |======================


OpenVINO 2024.5
Model: Road Segmentation ADAS FP16-INT8 - Device: CPU
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 1024.28 |=====================


OpenVINO 2024.5
Model: Road Segmentation ADAS FP16-INT8 - Device: CPU
FPS > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 31.04 |=======================


OpenVINO 2024.5
Model: Face Detection Retail FP16-INT8 - Device: CPU
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 81.47 |=======================


OpenVINO 2024.5
Model: Face Detection Retail FP16-INT8 - Device: CPU
FPS > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 392.26 |======================


OpenVINO 2024.5
Model: Weld Porosity Detection FP16 - Device: CPU
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 64.03 |=======================


OpenVINO 2024.5
Model: Weld Porosity Detection FP16 - Device: CPU
FPS > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 499.46 |======================


OpenVINO 2024.5
Model: Vehicle Detection FP16-INT8 - Device: CPU
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 370.01 |======================


OpenVINO 2024.5
Model: Vehicle Detection FP16-INT8 - Device: CPU
FPS > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 86.31 |=======================


OpenVINO 2024.5
Model: Road Segmentation ADAS FP16 - Device: CPU
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 137.21 |======================


OpenVINO 2024.5
Model: Road Segmentation ADAS FP16 - Device: CPU
FPS > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 232.83 |======================


OpenVINO 2024.5
Model: Face Detection FP16-INT8 - Device: CPU
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 4789.33 |=====================


OpenVINO 2024.5
Model: Face Detection FP16-INT8 - Device: CPU
FPS > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 6.53 |========================


OpenVINO 2024.5
Model: Vehicle Detection FP16 - Device: CPU
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 60.15 |=======================


OpenVINO 2024.5
Model: Vehicle Detection FP16 - Device: CPU
FPS > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 531.69 |======================


OpenVINO 2024.5
Model: Person Detection FP32 - Device: CPU
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 937.25 |======================


OpenVINO 2024.5
Model: Person Detection FP32 - Device: CPU
FPS > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 34.03 |=======================


OpenVINO 2024.5
Model: Person Detection FP16 - Device: CPU
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 937.64 |======================


OpenVINO 2024.5
Model: Person Detection FP16 - Device: CPU
FPS > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 34.02 |=======================


OpenVINO 2024.5
Model: Face Detection FP16 - Device: CPU
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 4566.75 |=====================


OpenVINO 2024.5
Model: Face Detection FP16 - Device: CPU
FPS > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 6.90 |========================


OpenVINO GenAI 2024.5
Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU - Time Per Output Token
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 99.0 |========================


OpenVINO GenAI 2024.5
Model: Phi-3-mini-128k-instruct-int4-ov - Device: CPU - Time To First Token
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 125.29 |======================


OpenVINO GenAI 2024.5
Model: Falcon-7b-instruct-int4-ov - Device: CPU - Time Per Output Token
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 101.57 |======================


OpenVINO GenAI 2024.5
Model: Falcon-7b-instruct-int4-ov - Device: CPU - Time To First Token
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 121.94 |======================


OpenVINO GenAI 2024.5
Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPU - Time Per Output Token
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 50.72 |=======================


OpenVINO GenAI 2024.5
Model: TinyLlama-1.1B-Chat-v1.0 - Device: CPU - Time To First Token
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 72.65 |=======================


OpenVINO GenAI 2024.5
Model: Gemma-7b-int4-ov - Device: CPU - Time Per Output Token
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 166.58 |======================


OpenVINO GenAI 2024.5
Model: Gemma-7b-int4-ov - Device: CPU - Time To First Token
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 198.52 |======================


OpenVINO 2024.5
Model: Face Detection Retail FP16 - Device: CPU
ms < Lower Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 27.94 |=======================


OpenVINO 2024.5
Model: Face Detection Retail FP16 - Device: CPU
FPS > Higher Is Better
ARMv8 Neoverse-N1 - NVIDIA RTX A400/PCIe 4GB - . 1153.54 |=====================