deepsparse 1.7 raptor lake Intel Core i9-14900K testing with a ASUS PRIME Z790-P WIFI (1402 BIOS) and ASUS Intel RPL-S 31GB on Ubuntu 23.10 via the Phoronix Test Suite. a: Processor: Intel Core i9-14900K @ 5.70GHz (24 Cores / 32 Threads), Motherboard: ASUS PRIME Z790-P WIFI (1402 BIOS), Chipset: Intel Device 7a27, Memory: 2 x 16GB DRAM-6000MT/s Corsair CMK32GX5M2B6000C36, Disk: Western Digital WD_BLACK SN850X 1000GB, Graphics: ASUS Intel RPL-S 31GB (1650MHz), Audio: Realtek ALC897, Monitor: ASUS VP28U OS: Ubuntu 23.10, Kernel: 6.8.0-phx (x86_64), Desktop: GNOME Shell 45.1, Display Server: X Server 1.21.1.7, OpenGL: 4.6 Mesa 24.0~git2312240600.c05261~oibaf~m (git-c05261a 2023-12-24 mantic-oibaf-ppa), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 3840x2160 b: Processor: Intel Core i9-14900K @ 5.70GHz (24 Cores / 32 Threads), Motherboard: ASUS PRIME Z790-P WIFI (1402 BIOS), Chipset: Intel Device 7a27, Memory: 2 x 16GB DRAM-6000MT/s Corsair CMK32GX5M2B6000C36, Disk: Western Digital WD_BLACK SN850X 1000GB, Graphics: ASUS Intel RPL-S 31GB (1650MHz), Audio: Realtek ALC897, Monitor: ASUS VP28U OS: Ubuntu 23.10, Kernel: 6.8.0-phx (x86_64), Desktop: GNOME Shell 45.1, Display Server: X Server 1.21.1.7, OpenGL: 4.6 Mesa 24.0~git2312240600.c05261~oibaf~m (git-c05261a 2023-12-24 mantic-oibaf-ppa), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 3840x2160 c: Processor: Intel Core i9-14900K @ 5.70GHz (24 Cores / 32 Threads), Motherboard: ASUS PRIME Z790-P WIFI (1402 BIOS), Chipset: Intel Device 7a27, Memory: 2 x 16GB DRAM-6000MT/s Corsair CMK32GX5M2B6000C36, Disk: Western Digital WD_BLACK SN850X 1000GB, Graphics: ASUS Intel RPL-S 31GB (1650MHz), Audio: Realtek ALC897, Monitor: ASUS VP28U OS: Ubuntu 23.10, Kernel: 6.8.0-phx (x86_64), Desktop: GNOME Shell 45.1, Display Server: X Server 1.21.1.7, OpenGL: 4.6 Mesa 24.0~git2312240600.c05261~oibaf~m (git-c05261a 2023-12-24 mantic-oibaf-ppa), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 3840x2160 d: Processor: Intel Core i9-14900K @ 5.70GHz (24 Cores / 32 Threads), Motherboard: ASUS PRIME Z790-P WIFI (1402 BIOS), Chipset: Intel Device 7a27, Memory: 2 x 16GB DRAM-6000MT/s Corsair CMK32GX5M2B6000C36, Disk: Western Digital WD_BLACK SN850X 1000GB, Graphics: ASUS Intel RPL-S 31GB (1650MHz), Audio: Realtek ALC897, Monitor: ASUS VP28U OS: Ubuntu 23.10, Kernel: 6.8.0-phx (x86_64), Desktop: GNOME Shell 45.1, Display Server: X Server 1.21.1.7, OpenGL: 4.6 Mesa 24.0~git2312240600.c05261~oibaf~m (git-c05261a 2023-12-24 mantic-oibaf-ppa), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 3840x2160 e: Processor: Intel Core i9-14900K @ 5.70GHz (24 Cores / 32 Threads), Motherboard: ASUS PRIME Z790-P WIFI (1402 BIOS), Chipset: Intel Device 7a27, Memory: 2 x 16GB DRAM-6000MT/s Corsair CMK32GX5M2B6000C36, Disk: Western Digital WD_BLACK SN850X 1000GB, Graphics: ASUS Intel RPL-S 31GB (1650MHz), Audio: Realtek ALC897, Monitor: ASUS VP28U OS: Ubuntu 23.10, Kernel: 6.8.0-phx (x86_64), Desktop: GNOME Shell 45.1, Display Server: X Server 1.21.1.7, OpenGL: 4.6 Mesa 24.0~git2312240600.c05261~oibaf~m (git-c05261a 2023-12-24 mantic-oibaf-ppa), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 3840x2160 Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 975.81 |================================================================ b . 941.79 |============================================================= c . 927.77 |============================================================= d . 1011.71 |================================================================== e . 994.77 |================================================================= Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 1.0225 |================================================================ b . 1.0591 |================================================================== c . 1.0746 |=================================================================== d . 0.9856 |============================================================= e . 1.0024 |============================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 3.4607 |================================================================== b . 3.4488 |================================================================= c . 3.5300 |=================================================================== d . 3.4601 |================================================================== e . 3.4336 |================================================================= Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 1152.25 |================================================================= b . 1156.18 |================================================================== c . 1129.73 |================================================================ d . 1152.75 |================================================================== e . 1161.30 |================================================================== Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 7.6693 |================================================================== b . 7.7518 |=================================================================== c . 7.5635 |================================================================= d . 7.6435 |================================================================== e . 7.6128 |================================================================== Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 518.25 |================================================================== b . 512.69 |================================================================= c . 525.39 |=================================================================== d . 520.01 |================================================================== e . 522.00 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 20.08 |==================================================================== b . 19.66 |=================================================================== c . 19.65 |=================================================================== d . 19.82 |=================================================================== e . 19.96 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 49.79 |=================================================================== b . 50.85 |==================================================================== c . 50.89 |==================================================================== d . 50.44 |=================================================================== e . 50.09 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 72.99 |==================================================================== b . 71.43 |=================================================================== c . 71.44 |=================================================================== d . 72.07 |=================================================================== e . 71.94 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 13.70 |=================================================================== b . 14.00 |==================================================================== c . 14.00 |==================================================================== d . 13.87 |=================================================================== e . 13.90 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 374.66 |=================================================================== b . 368.59 |================================================================== c . 373.12 |=================================================================== d . 375.63 |=================================================================== e . 368.23 |================================================================== Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 10.68 |=================================================================== b . 10.85 |==================================================================== c . 10.72 |=================================================================== d . 10.65 |=================================================================== e . 10.86 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 8.0391 |=================================================================== b . 7.8904 |================================================================== c . 7.9405 |================================================================== d . 7.9734 |================================================================== e . 7.9248 |================================================================== Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 124.33 |================================================================== b . 126.67 |=================================================================== c . 125.87 |=================================================================== d . 125.35 |================================================================== e . 126.11 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 104.12 |================================================================== b . 104.39 |================================================================== c . 105.19 |=================================================================== d . 105.54 |=================================================================== e . 104.52 |================================================================== Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 9.6041 |=================================================================== b . 9.5789 |=================================================================== c . 9.5059 |================================================================== d . 9.4750 |================================================================== e . 9.5673 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 27.97 |==================================================================== b . 27.79 |==================================================================== c . 27.90 |==================================================================== d . 27.91 |==================================================================== e . 27.62 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 105.88 |================================================================== b . 106.21 |=================================================================== c . 105.44 |================================================================== d . 106.80 |=================================================================== e . 106.63 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 142.92 |================================================================== b . 143.84 |=================================================================== c . 143.29 |================================================================== d . 143.24 |================================================================== e . 144.75 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 9.4444 |=================================================================== b . 9.4148 |=================================================================== c . 9.4838 |=================================================================== d . 9.3639 |================================================================== e . 9.3782 |================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 324.56 |================================================================== b . 327.19 |=================================================================== c . 326.89 |=================================================================== d . 327.29 |=================================================================== e . 328.56 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 3.0776 |=================================================================== b . 3.0531 |================================================================== c . 3.0560 |=================================================================== d . 3.0521 |================================================================== e . 3.0404 |================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 7.9967 |=================================================================== b . 7.9282 |================================================================== c . 7.9467 |=================================================================== d . 7.9961 |=================================================================== e . 7.9452 |=================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 124.99 |================================================================== b . 126.07 |=================================================================== c . 125.77 |=================================================================== d . 125.00 |================================================================== e . 125.80 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 269.97 |=================================================================== b . 268.08 |=================================================================== c . 268.00 |=================================================================== d . 269.62 |=================================================================== e . 268.55 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 14.82 |==================================================================== b . 14.92 |==================================================================== c . 14.92 |==================================================================== d . 14.83 |==================================================================== e . 14.89 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 81.49 |==================================================================== b . 81.50 |==================================================================== c . 81.51 |==================================================================== d . 81.68 |==================================================================== e . 81.17 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 49.07 |==================================================================== b . 49.07 |==================================================================== c . 49.06 |==================================================================== d . 48.96 |==================================================================== e . 49.26 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 379.71 |=================================================================== b . 379.14 |=================================================================== c . 378.26 |=================================================================== d . 378.17 |=================================================================== e . 377.36 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 10.53 |==================================================================== b . 10.55 |==================================================================== c . 10.57 |==================================================================== d . 10.58 |==================================================================== e . 10.60 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 9.1560 |=================================================================== b . 9.1990 |=================================================================== c . 9.1596 |=================================================================== d . 9.1642 |=================================================================== e . 9.2062 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 436.25 |=================================================================== b . 434.22 |=================================================================== c . 436.12 |=================================================================== d . 435.85 |=================================================================== e . 433.88 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 67.26 |==================================================================== b . 67.62 |==================================================================== c . 67.38 |==================================================================== d . 67.41 |==================================================================== e . 67.32 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 14.86 |==================================================================== b . 14.78 |==================================================================== c . 14.84 |==================================================================== d . 14.83 |==================================================================== e . 14.85 |==================================================================== Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 20.33 |==================================================================== b . 20.24 |==================================================================== c . 20.26 |==================================================================== d . 20.27 |==================================================================== e . 20.23 |==================================================================== Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 196.62 |=================================================================== b . 197.48 |=================================================================== c . 197.28 |=================================================================== d . 197.20 |=================================================================== e . 197.57 |=================================================================== Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 7.9348 |=================================================================== b . 7.9643 |=================================================================== c . 7.9518 |=================================================================== d . 7.9571 |=================================================================== e . 7.9673 |=================================================================== Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 125.92 |=================================================================== b . 125.45 |=================================================================== c . 125.63 |=================================================================== d . 125.57 |=================================================================== e . 125.41 |=================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 27.95 |==================================================================== b . 27.92 |==================================================================== c . 27.88 |==================================================================== d . 27.96 |==================================================================== e . 27.85 |==================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 143.04 |=================================================================== b . 143.19 |=================================================================== c . 143.39 |=================================================================== d . 143.00 |=================================================================== e . 143.57 |=================================================================== Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 88.87 |==================================================================== b . 88.87 |==================================================================== c . 88.94 |==================================================================== d . 88.93 |==================================================================== e . 88.98 |==================================================================== Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 11.25 |==================================================================== b . 11.25 |==================================================================== c . 11.24 |==================================================================== d . 11.24 |==================================================================== e . 11.24 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 71.70 |==================================================================== b . 71.75 |==================================================================== c . 71.76 |==================================================================== d . 71.76 |==================================================================== e . 71.78 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 55.77 |==================================================================== b . 55.74 |==================================================================== c . 55.72 |==================================================================== d . 55.73 |==================================================================== e . 55.71 |====================================================================