deepsparse unv Intel Core i7-1165G7 testing with a Dell 0GG9PT (3.11.0 BIOS) and Intel Xe TGL GT2 15GB on Ubuntu 22.10 via the Phoronix Test Suite. a: Processor: Intel Core i7-1165G7 @ 4.70GHz (4 Cores / 8 Threads), Motherboard: Dell 0GG9PT (3.11.0 BIOS), Chipset: Intel Tiger Lake-LP, Memory: 16GB, Disk: Kioxia KBG40ZNS256G NVMe 256GB, Graphics: Intel Xe TGL GT2 15GB (1300MHz), Audio: Realtek ALC289, Network: Intel Wi-Fi 6 AX201 OS: Ubuntu 22.10, Kernel: 5.19.0-29-generic (x86_64), Desktop: GNOME Shell 43.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 22.2.1, Vulkan: 1.3.224, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1200 b: Processor: Intel Core i7-1165G7 @ 4.70GHz (4 Cores / 8 Threads), Motherboard: Dell 0GG9PT (3.11.0 BIOS), Chipset: Intel Tiger Lake-LP, Memory: 16GB, Disk: Kioxia KBG40ZNS256G NVMe 256GB, Graphics: Intel Xe TGL GT2 15GB (1300MHz), Audio: Realtek ALC289, Network: Intel Wi-Fi 6 AX201 OS: Ubuntu 22.10, Kernel: 5.19.0-29-generic (x86_64), Desktop: GNOME Shell 43.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 22.2.1, Vulkan: 1.3.224, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1200 c: Processor: Intel Core i7-1165G7 @ 4.70GHz (4 Cores / 8 Threads), Motherboard: Dell 0GG9PT (3.11.0 BIOS), Chipset: Intel Tiger Lake-LP, Memory: 16GB, Disk: Kioxia KBG40ZNS256G NVMe 256GB, Graphics: Intel Xe TGL GT2 15GB (1300MHz), Audio: Realtek ALC289, Network: Intel Wi-Fi 6 AX201 OS: Ubuntu 22.10, Kernel: 5.19.0-29-generic (x86_64), Desktop: GNOME Shell 43.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 22.2.1, Vulkan: 1.3.224, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1200 d: Processor: Intel Core i7-1165G7 @ 4.70GHz (4 Cores / 8 Threads), Motherboard: Dell 0GG9PT (3.11.0 BIOS), Chipset: Intel Tiger Lake-LP, Memory: 16GB, Disk: Kioxia KBG40ZNS256G NVMe 256GB, Graphics: Intel Xe TGL GT2 15GB (1300MHz), Audio: Realtek ALC289, Network: Intel Wi-Fi 6 AX201 OS: Ubuntu 22.10, Kernel: 5.19.0-29-generic (x86_64), Desktop: GNOME Shell 43.0, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 22.2.1, Vulkan: 1.3.224, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1200 Unvanquished 0.54 Resolution: 1920 x 1080 - Effects Quality: High Frames Per Second > Higher Is Better a . 288.9 |==================================================================== b . 289.8 |==================================================================== c . 287.4 |=================================================================== d . 284.4 |=================================================================== Unvanquished 0.54 Resolution: 1920 x 1200 - Effects Quality: High Frames Per Second > Higher Is Better a . 282.8 |==================================================================== b . 283.2 |==================================================================== c . 282.0 |==================================================================== d . 279.0 |=================================================================== Unvanquished 0.54 Resolution: 1920 x 1080 - Effects Quality: Ultra Frames Per Second > Higher Is Better a . 218.9 |==================================================================== b . 220.1 |==================================================================== c . 217.0 |=================================================================== d . 216.0 |=================================================================== Unvanquished 0.54 Resolution: 1920 x 1200 - Effects Quality: Ultra Frames Per Second > Higher Is Better a . 208.8 |==================================================================== b . 208.2 |==================================================================== c . 207.0 |=================================================================== d . 205.3 |=================================================================== Unvanquished 0.54 Resolution: 1920 x 1080 - Effects Quality: Medium Frames Per Second > Higher Is Better a . 313.5 |=================================================================== b . 318.2 |==================================================================== c . 313.3 |=================================================================== d . 311.8 |=================================================================== Unvanquished 0.54 Resolution: 1920 x 1200 - Effects Quality: Medium Frames Per Second > Higher Is Better a . 308.1 |================================================================== b . 315.2 |==================================================================== c . 310.5 |=================================================================== d . 308.8 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 3.6314 |=================================================================== b . 3.6170 |=================================================================== c . 3.1630 |========================================================== d . 3.2394 |============================================================ Neural Magic DeepSparse 1.3.2 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 550.73 |========================================================== b . 552.87 |========================================================== c . 634.17 |=================================================================== d . 619.92 |================================================================= Neural Magic DeepSparse 1.3.2 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 3.5177 |=================================================================== b . 3.5154 |=================================================================== c . 3.1185 |=========================================================== d . 3.1391 |============================================================ Neural Magic DeepSparse 1.3.2 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 284.26 |=========================================================== b . 284.45 |=========================================================== c . 321.17 |=================================================================== d . 319.18 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 68.61 |=================================================================== b . 69.85 |==================================================================== c . 64.58 |=============================================================== d . 62.33 |============================================================= Neural Magic DeepSparse 1.3.2 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 29.12 |============================================================== b . 28.60 |============================================================= c . 31.00 |================================================================== d . 32.05 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 62.16 |==================================================================== b . 59.99 |================================================================== c . 58.21 |================================================================ d . 57.21 |=============================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 16.08 |============================================================== b . 16.66 |================================================================= c . 17.21 |=================================================================== d . 17.50 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 17.62 |================================================================== b . 18.17 |==================================================================== c . 15.73 |=========================================================== d . 15.64 |=========================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 113.47 |=========================================================== b . 110.00 |========================================================== c . 127.22 |=================================================================== d . 127.84 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 17.01 |=================================================================== b . 17.31 |==================================================================== c . 16.11 |=============================================================== d . 15.93 |=============================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 58.79 |================================================================ b . 57.77 |=============================================================== c . 62.15 |=================================================================== d . 62.79 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 21.81 |=================================================================== b . 22.20 |==================================================================== c . 20.89 |================================================================ d . 20.46 |=============================================================== Neural Magic DeepSparse 1.3.2 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 91.65 |================================================================ b . 90.07 |=============================================================== c . 95.96 |=================================================================== d . 97.90 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 20.59 |==================================================================== b . 20.61 |==================================================================== c . 18.42 |============================================================= d . 17.97 |=========================================================== Neural Magic DeepSparse 1.3.2 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 48.54 |=========================================================== b . 48.50 |=========================================================== c . 54.54 |================================================================== d . 55.79 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 58.78 |================================================================== b . 60.16 |==================================================================== c . 52.02 |=========================================================== d . 50.64 |========================================================= Neural Magic DeepSparse 1.3.2 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 34.00 |=========================================================== b . 33.21 |========================================================= c . 38.48 |================================================================== d . 39.50 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 49.51 |=================================================================== b . 49.68 |=================================================================== c . 50.10 |==================================================================== d . 49.85 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 20.19 |==================================================================== b . 20.12 |==================================================================== c . 19.99 |=================================================================== d . 20.08 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 31.64 |=================================================================== b . 32.24 |==================================================================== c . 28.57 |============================================================ d . 27.88 |=========================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 63.18 |============================================================ b . 62.02 |=========================================================== c . 70.38 |================================================================== d . 72.03 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 30.03 |================================================================== b . 30.74 |==================================================================== c . 27.32 |============================================================ d . 26.53 |=========================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 33.29 |============================================================ b . 32.53 |=========================================================== c . 36.80 |================================================================== d . 37.81 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 6.5581 |============================================================ b . 7.3184 |=================================================================== c . 6.7233 |============================================================== d . 6.6054 |============================================================ Neural Magic DeepSparse 1.3.2 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 304.93 |=================================================================== b . 273.22 |============================================================ c . 297.54 |================================================================= d . 302.30 |================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 6.9658 |=================================================================== b . 6.5168 |=============================================================== c . 6.6878 |================================================================ d . 6.5320 |=============================================================== Neural Magic DeepSparse 1.3.2 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 143.54 |=============================================================== b . 153.43 |=================================================================== c . 149.68 |================================================================= d . 153.07 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 14.59 |================================================================ b . 15.58 |==================================================================== c . 14.11 |============================================================== d . 13.76 |============================================================ Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 137.06 |=============================================================== b . 128.37 |=========================================================== c . 142.24 |================================================================= d . 145.71 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 13.59 |================================================================= b . 14.16 |==================================================================== c . 13.15 |=============================================================== d . 13.01 |============================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 73.57 |================================================================= b . 70.60 |============================================================== c . 76.24 |=================================================================== d . 77.00 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 3.6554 |=================================================================== b . 3.1909 |========================================================== c . 3.1323 |========================================================= d . 3.0903 |========================================================= Neural Magic DeepSparse 1.3.2 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 547.10 |========================================================= b . 626.70 |================================================================= c . 639.68 |================================================================== d . 648.16 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 3.5135 |=================================================================== b . 3.0315 |========================================================== c . 3.1354 |============================================================ d . 3.0976 |=========================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 284.61 |========================================================== b . 329.87 |=================================================================== c . 319.51 |================================================================= d . 323.23 |==================================================================