ddddeep Intel Core i7-1280P testing with a MSI Prestige 14Evo A12M MS-14C6 (E14C6IMS.115 BIOS) and MSI Intel ADL GT2 8GB on Ubuntu 23.10 via the Phoronix Test Suite. a: Processor: Intel Core i7-1280P @ 4.70GHz (14 Cores / 20 Threads), Motherboard: MSI Prestige 14Evo A12M MS-14C6 (E14C6IMS.115 BIOS), Chipset: Intel Alder Lake PCH, Memory: 8 x 2GB LPDDR4-4267MT/s SK Hynix H9HCNNNCPMMLXR-, Disk: 1024GB Micron_3400_MTFDKBA1T0TFH, Graphics: MSI Intel ADL GT2 8GB (1450MHz), Audio: Realtek ALC274, Network: Intel Alder Lake-P PCH CNVi WiFi OS: Ubuntu 23.10, Kernel: 6.7.0-060700-generic (x86_64), Desktop: GNOME Shell 45.2, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.1~git2401210600.c3a64f~oibaf~m (git-c3a64f8 2024-01-21 mantic-oibaf-ppa), OpenCL: OpenCL 3.0, Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080 b: Processor: Intel Core i7-1280P @ 4.70GHz (14 Cores / 20 Threads), Motherboard: MSI Prestige 14Evo A12M MS-14C6 (E14C6IMS.115 BIOS), Chipset: Intel Alder Lake PCH, Memory: 8 x 2GB LPDDR4-4267MT/s SK Hynix H9HCNNNCPMMLXR-, Disk: 1024GB Micron_3400_MTFDKBA1T0TFH, Graphics: MSI Intel ADL GT2 8GB (1450MHz), Audio: Realtek ALC274, Network: Intel Alder Lake-P PCH CNVi WiFi OS: Ubuntu 23.10, Kernel: 6.7.0-060700-generic (x86_64), Desktop: GNOME Shell 45.2, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.1~git2401210600.c3a64f~oibaf~m (git-c3a64f8 2024-01-21 mantic-oibaf-ppa), OpenCL: OpenCL 3.0, Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080 c: Processor: Intel Core i7-1280P @ 4.70GHz (14 Cores / 20 Threads), Motherboard: MSI Prestige 14Evo A12M MS-14C6 (E14C6IMS.115 BIOS), Chipset: Intel Alder Lake PCH, Memory: 8 x 2GB LPDDR4-4267MT/s SK Hynix H9HCNNNCPMMLXR-, Disk: 1024GB Micron_3400_MTFDKBA1T0TFH, Graphics: MSI Intel ADL GT2 8GB (1450MHz), Audio: Realtek ALC274, Network: Intel Alder Lake-P PCH CNVi WiFi OS: Ubuntu 23.10, Kernel: 6.7.0-060700-generic (x86_64), Desktop: GNOME Shell 45.2, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 24.1~git2401210600.c3a64f~oibaf~m (git-c3a64f8 2024-01-21 mantic-oibaf-ppa), OpenCL: OpenCL 3.0, Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080 Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 4.6509 |============================================================== b . 4.8027 |================================================================ c . 5.0006 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 644.99 |=================================================================== b . 624.61 |================================================================= c . 598.35 |============================================================== Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 4.2542 |================================================================= b . 3.9587 |============================================================= c . 4.3521 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 235.05 |============================================================== b . 252.60 |=================================================================== c . 229.76 |============================================================= Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 162.04 |================================================================== b . 150.84 |============================================================= c . 164.63 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 18.48 |=============================================================== b . 19.85 |==================================================================== c . 18.19 |============================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 128.81 |================================================================== b . 121.94 |============================================================== c . 130.99 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 7.7576 |=============================================================== b . 8.1944 |=================================================================== c . 7.6286 |============================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 52.96 |================================================================= b . 53.72 |================================================================== c . 55.00 |==================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 56.59 |==================================================================== b . 55.78 |=================================================================== c . 54.51 |================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 46.11 |=============================================================== b . 46.81 |================================================================ c . 49.48 |==================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 21.68 |==================================================================== b . 21.36 |=================================================================== c . 20.20 |=============================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 403.88 |============================================================== b . 412.70 |================================================================ c . 433.83 |=================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 7.3983 |=================================================================== b . 7.2381 |================================================================== c . 6.8862 |============================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 335.40 |================================================================== b . 334.58 |================================================================= c . 342.85 |=================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 2.9712 |=================================================================== b . 2.9789 |=================================================================== c . 2.9071 |================================================================= Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 4.6331 |================================================================== b . 4.6873 |=================================================================== c . 4.6837 |=================================================================== Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 642.86 |=================================================================== b . 635.47 |================================================================== c . 635.97 |================================================================== Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 6.8850 |=================================================================== b . 6.7827 |================================================================== c . 6.8513 |=================================================================== Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 145.22 |================================================================== b . 147.41 |=================================================================== c . 145.94 |================================================================== Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 52.71 |=================================================================== b . 51.94 |================================================================== c . 53.50 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 56.88 |=================================================================== b . 57.73 |==================================================================== c . 56.04 |================================================================== Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 43.76 |=================================================================== b . 44.36 |==================================================================== c . 44.47 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 22.84 |==================================================================== b . 22.54 |=================================================================== c . 22.48 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 24.53 |==================================================================== b . 22.56 |============================================================== c . 24.63 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 122.27 |============================================================== b . 132.91 |=================================================================== c . 121.76 |============================================================= Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 24.79 |=================================================================== b . 24.94 |==================================================================== c . 24.99 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 40.33 |==================================================================== b . 40.09 |==================================================================== c . 40.01 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 35.74 |================================================================ b . 38.09 |==================================================================== c . 35.75 |================================================================ Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 83.90 |==================================================================== b . 78.74 |================================================================ c . 83.89 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 32.08 |================================================================= b . 33.82 |==================================================================== c . 31.79 |================================================================ Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 31.17 |=================================================================== b . 29.57 |================================================================ c . 31.46 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 4.9333 |=============================================================== b . 5.2179 |=================================================================== c . 4.9986 |================================================================ Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 607.78 |=================================================================== b . 574.90 |=============================================================== c . 600.12 |================================================================== Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 5.0335 |=================================================================== b . 4.7566 |=============================================================== c . 5.0119 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 198.65 |=============================================================== b . 210.21 |=================================================================== c . 199.51 |================================================================ Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 65.59 |========================================================= b . 61.69 |====================================================== c . 77.60 |==================================================================== Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 45.70 |================================================================ b . 48.59 |==================================================================== c . 38.61 |====================================================== Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 51.00 |==================================================================== b . 48.96 |================================================================= c . 51.24 |==================================================================== Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 19.60 |================================================================= b . 20.41 |==================================================================== c . 19.51 |================================================================= Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 3.8107 |=============================================================== b . 4.0838 |=================================================================== c . 3.7499 |============================================================== Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 778.50 |================================================================== b . 730.02 |============================================================== c . 793.88 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 4.0998 |=================================================================== b . 3.7941 |============================================================== c . 4.0441 |================================================================== Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 243.91 |============================================================== b . 263.56 |=================================================================== c . 247.27 |===============================================================