deepsparse tigerlake Intel Core i7-1165G7 testing with a Dell 0GG9PT (3.3.0 BIOS) and Intel Xe TGL GT2 3GB on Ubuntu 21.10 via the Phoronix Test Suite. A: Processor: Intel Core i7-1165G7 @ 4.70GHz (4 Cores / 8 Threads), Motherboard: Dell 0GG9PT (3.3.0 BIOS), Chipset: Intel Tiger Lake-LP, Memory: 16GB, Disk: Kioxia KBG40ZNS256G NVMe 256GB, Graphics: Intel Xe TGL GT2 3GB (1300MHz), Audio: Realtek ALC289, Network: Intel Wi-Fi 6 AX201 OS: Ubuntu 21.10, Kernel: 5.13.0-52-generic (x86_64), Desktop: GNOME Shell 40.5, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 21.2.2, Vulkan: 1.2.182, Compiler: GCC 11.2.0, File-System: ext4, Screen Resolution: 1920x1200 B: Processor: Intel Core i7-1165G7 @ 4.70GHz (4 Cores / 8 Threads), Motherboard: Dell 0GG9PT (3.3.0 BIOS), Chipset: Intel Tiger Lake-LP, Memory: 16GB, Disk: Kioxia KBG40ZNS256G NVMe 256GB, Graphics: Intel Xe TGL GT2 3GB (1300MHz), Audio: Realtek ALC289, Network: Intel Wi-Fi 6 AX201 OS: Ubuntu 21.10, Kernel: 5.13.0-52-generic (x86_64), Desktop: GNOME Shell 40.5, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 21.2.2, Vulkan: 1.2.182, Compiler: GCC 11.2.0, File-System: ext4, Screen Resolution: 1920x1200 C: Processor: Intel Core i7-1165G7 @ 4.70GHz (4 Cores / 8 Threads), Motherboard: Dell 0GG9PT (3.3.0 BIOS), Chipset: Intel Tiger Lake-LP, Memory: 16GB, Disk: Kioxia KBG40ZNS256G NVMe 256GB, Graphics: Intel Xe TGL GT2 3GB (1300MHz), Audio: Realtek ALC289, Network: Intel Wi-Fi 6 AX201 OS: Ubuntu 21.10, Kernel: 5.13.0-52-generic (x86_64), Desktop: GNOME Shell 40.5, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 21.2.2, Vulkan: 1.2.182, Compiler: GCC 11.2.0, File-System: ext4, Screen Resolution: 1920x1200 Neural Magic DeepSparse 1.1 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better A . 522.36 |====================================================== B . 636.17 |================================================================= C . 652.82 |=================================================================== Neural Magic DeepSparse 1.1 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better A . 3.8206 |=================================================================== B . 3.1607 |======================================================= C . 3.0716 |====================================================== Neural Magic DeepSparse 1.1 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better A . 133.82 |============================================================ B . 147.81 |=================================================================== C . 148.66 |=================================================================== Neural Magic DeepSparse 1.1 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better A . 14.94 |==================================================================== B . 13.56 |============================================================== C . 13.48 |============================================================= Neural Magic DeepSparse 1.1 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better A . 77.94 |==================================================================== B . 76.28 |=================================================================== C . 75.17 |================================================================== Neural Magic DeepSparse 1.1 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better A . 12.83 |================================================================= B . 13.13 |=================================================================== C . 13.35 |==================================================================== Neural Magic DeepSparse 1.1 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better A . 331.67 |=================================================================== B . 325.08 |================================================================== C . 324.58 |================================================================== Neural Magic DeepSparse 1.1 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better A . 3.0149 |================================================================= B . 3.0819 |=================================================================== C . 3.0857 |=================================================================== Neural Magic DeepSparse 1.1 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better A . 65.26 |============================================================= B . 72.51 |==================================================================== C . 72.74 |==================================================================== Neural Magic DeepSparse 1.1 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better A . 30.63 |==================================================================== B . 27.74 |============================================================== C . 27.59 |============================================================= Neural Magic DeepSparse 1.1 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better A . 39.38 |==================================================================== B . 37.81 |================================================================= C . 38.40 |================================================================== Neural Magic DeepSparse 1.1 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream items/sec > Higher Is Better A . 25.39 |================================================================= B . 26.60 |==================================================================== C . 26.14 |=================================================================== Neural Magic DeepSparse 1.1 Model: CV Detection,YOLOv5s COCO - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better A . 81.40 |========================================================= B . 96.33 |==================================================================== C . 95.77 |==================================================================== Neural Magic DeepSparse 1.1 Model: CV Detection,YOLOv5s COCO - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better A . 24.56 |==================================================================== B . 20.78 |========================================================== C . 20.92 |========================================================== Neural Magic DeepSparse 1.1 Model: CV Detection,YOLOv5s COCO - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better A . 53.27 |==================================================================== B . 52.54 |=================================================================== C . 52.17 |=================================================================== Neural Magic DeepSparse 1.1 Model: CV Detection,YOLOv5s COCO - Scenario: Synchronous Single-Stream items/sec > Higher Is Better A . 18.77 |================================================================== B . 19.07 |=================================================================== C . 19.23 |==================================================================== Neural Magic DeepSparse 1.1 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better A . 18.68 |=============================================================== B . 20.04 |==================================================================== C . 19.85 |=================================================================== Neural Magic DeepSparse 1.1 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream items/sec > Higher Is Better A . 53.50 |==================================================================== B . 49.95 |=============================================================== C . 50.50 |================================================================ Neural Magic DeepSparse 1.1 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better A . 602.20 |============================================================ B . 672.76 |=================================================================== C . 645.27 |================================================================ Neural Magic DeepSparse 1.1 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better A . 3.3209 |=================================================================== B . 2.9727 |============================================================ C . 3.1063 |=============================================================== Neural Magic DeepSparse 1.1 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better A . 283.16 |========================================================= B . 323.62 |================================================================= C . 332.04 |=================================================================== Neural Magic DeepSparse 1.1 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream items/sec > Higher Is Better A . 3.5314 |=================================================================== B . 3.0965 |=========================================================== C . 3.0117 |========================================================= Neural Magic DeepSparse 1.1 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better A . 55.85 |============================================================= B . 62.71 |==================================================================== C . 62.31 |==================================================================== Neural Magic DeepSparse 1.1 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better A . 17.90 |==================================================================== B . 15.94 |============================================================= C . 16.06 |============================================================= Neural Magic DeepSparse 1.1 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better A . 34.79 |============================================================ B . 39.14 |=================================================================== C . 39.43 |==================================================================== Neural Magic DeepSparse 1.1 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better A . 57.43 |==================================================================== B . 51.11 |============================================================= C . 50.66 |============================================================ Neural Magic DeepSparse 1.1 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better A . 110.52 |========================================================== B . 127.87 |=================================================================== C . 126.97 |=================================================================== Neural Magic DeepSparse 1.1 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better A . 18.07 |==================================================================== B . 15.63 |=========================================================== C . 15.75 |===========================================================