a Benchmarks for a future article. a: Processor: 2 x INTEL XEON PLATINUM 8592+ @ 3.90GHz (128 Cores / 256 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3B05.TEL4P1 BIOS), Chipset: Intel Device 1bce, Memory: 1008GB, Disk: 3201GB Micron_7450_MTFDKCB3T2TFS, Graphics: ASPEED, Network: 2 x Intel X710 for 10GBASE-T OS: Ubuntu 23.10, Kernel: 6.5.0-13-generic (x86_64), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080 b: Processor: 2 x INTEL XEON PLATINUM 8592+ @ 3.90GHz (128 Cores / 256 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3B05.TEL4P1 BIOS), Chipset: Intel Device 1bce, Memory: 1008GB, Disk: 3201GB Micron_7450_MTFDKCB3T2TFS, Graphics: ASPEED, Network: 2 x Intel X710 for 10GBASE-T OS: Ubuntu 23.10, Kernel: 6.5.0-13-generic (x86_64), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080 c: Processor: 2 x INTEL XEON PLATINUM 8592+ @ 3.90GHz (128 Cores / 256 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3B05.TEL4P1 BIOS), Chipset: Intel Device 1bce, Memory: 1008GB, Disk: 3201GB Micron_7450_MTFDKCB3T2TFS, Graphics: ASPEED, Network: 2 x Intel X710 for 10GBASE-T OS: Ubuntu 23.10, Kernel: 6.5.0-13-generic (x86_64), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080 d: Processor: 2 x INTEL XEON PLATINUM 8592+ @ 3.90GHz (128 Cores / 256 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3B05.TEL4P1 BIOS), Chipset: Intel Device 1bce, Memory: 1008GB, Disk: 3201GB Micron_7450_MTFDKCB3T2TFS, Graphics: ASPEED, Network: 2 x Intel X710 for 10GBASE-T OS: Ubuntu 23.10, Kernel: 6.5.0-13-generic (x86_64), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080 Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 133.52 |=================================================================== b . 133.72 |=================================================================== c . 133.48 |=================================================================== d . 133.01 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 31.19 |================================================================ b . 31.23 |================================================================= c . 31.63 |================================================================= d . 32.89 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 4554.98 |================================================================== b . 4563.51 |================================================================== c . 4563.43 |================================================================== d . 4556.02 |================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 238.65 |=================================================================== b . 237.05 |================================================================== c . 230.76 |================================================================ d . 240.23 |=================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 1829.96 |================================================================== b . 1824.38 |================================================================== c . 1827.98 |================================================================== d . 1837.50 |================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 338.96 |=================================================================== b . 337.68 |=================================================================== c . 338.55 |=================================================================== d . 338.15 |=================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 11228.66 |================================================================= b . 11169.85 |================================================================= c . 11163.66 |================================================================= d . 11221.14 |================================================================= Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 1030.32 |================================================================= b . 1027.51 |================================================================= c . 1043.28 |================================================================== d . 1032.24 |================================================================= Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 828.04 |=================================================================== b . 828.65 |=================================================================== c . 828.45 |=================================================================== d . 825.08 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 190.84 |=================================================================== b . 190.04 |=================================================================== c . 189.82 |=================================================================== d . 190.14 |=================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 156.06 |=================================================================== b . 155.72 |=================================================================== c . 155.77 |=================================================================== d . 155.60 |=================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 29.94 |=================================================================== b . 30.39 |==================================================================== c . 30.29 |==================================================================== d . 29.52 |================================================================== Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 1832.79 |================================================================== b . 1817.99 |================================================================= c . 1834.48 |================================================================== d . 1819.69 |================================================================= Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 339.04 |=================================================================== b . 338.86 |=================================================================== c . 338.22 |=================================================================== d . 339.37 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 854.05 |=================================================================== b . 852.95 |=================================================================== c . 854.25 |=================================================================== d . 850.89 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 194.25 |=================================================================== b . 193.60 |=================================================================== c . 193.09 |=================================================================== d . 193.82 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 1231.20 |================================================================== b . 1231.29 |================================================================== c . 1231.27 |================================================================== d . 1226.80 |================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 199.69 |=================================================================== b . 199.34 |=================================================================== c . 199.33 |=================================================================== d . 196.79 |================================================================== Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 183.50 |=================================================================== b . 180.32 |================================================================== c . 177.15 |================================================================= d . 181.20 |================================================================== Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 34.96 |==================================================================== b . 35.11 |==================================================================== c . 35.01 |==================================================================== d . 35.00 |==================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 1880.77 |================================================================== b . 1875.90 |================================================================== c . 1879.84 |================================================================== d . 1865.02 |================================================================= Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 96.43 |==================================================================== b . 96.38 |==================================================================== c . 96.55 |==================================================================== d . 95.70 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 133.60 |=================================================================== b . 133.82 |=================================================================== c . 133.60 |=================================================================== d . 133.60 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 31.70 |================================================================= b . 31.53 |================================================================= c . 31.72 |================================================================= d . 33.05 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 475.70 |=================================================================== b . 474.19 |=================================================================== c . 475.68 |=================================================================== d . 477.61 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 32.06 |==================================================================== b . 32.04 |==================================================================== c . 31.65 |=================================================================== d . 30.42 |================================================================= Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 14.04 |==================================================================== b . 14.01 |==================================================================== c . 14.01 |==================================================================== d . 14.03 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 4.2083 |================================================================= b . 4.2295 |================================================================= c . 4.3436 |=================================================================== d . 4.1604 |================================================================ Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 34.92 |==================================================================== b . 35.01 |==================================================================== c . 34.97 |==================================================================== d . 34.76 |==================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 2.9492 |=================================================================== b . 2.9602 |=================================================================== c . 2.9530 |=================================================================== d . 2.9573 |=================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 5.6852 |=================================================================== b . 5.7140 |=================================================================== c . 5.7165 |=================================================================== d . 5.6908 |=================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 0.9681 |=================================================================== b . 0.9706 |=================================================================== c . 0.9556 |================================================================== d . 0.9667 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 77.18 |==================================================================== b . 77.09 |==================================================================== c . 77.12 |==================================================================== d . 77.46 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 5.2382 |=================================================================== b . 5.2617 |=================================================================== c . 5.2651 |=================================================================== d . 5.2589 |=================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 407.28 |=================================================================== b . 408.63 |=================================================================== c . 408.97 |=================================================================== d . 408.16 |=================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 33.40 |=================================================================== b . 32.90 |================================================================== c . 33.00 |================================================================== d . 33.86 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 34.87 |=================================================================== b . 35.14 |==================================================================== c . 34.82 |=================================================================== d . 35.10 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 2.9484 |=================================================================== b . 2.9500 |=================================================================== c . 2.9555 |=================================================================== d . 2.9455 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 74.82 |==================================================================== b . 74.94 |==================================================================== c . 74.79 |==================================================================== d . 75.08 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 5.1485 |=================================================================== b . 5.1659 |=================================================================== c . 5.1802 |=================================================================== d . 5.1614 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 51.93 |==================================================================== b . 51.93 |==================================================================== c . 51.92 |==================================================================== d . 52.07 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 5.0073 |================================================================== b . 5.0196 |================================================================== c . 5.0179 |================================================================== d . 5.0808 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 347.66 |================================================================= b . 353.69 |================================================================== c . 359.89 |=================================================================== d . 352.22 |================================================================== Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 28.59 |==================================================================== b . 28.47 |==================================================================== c . 28.55 |==================================================================== d . 28.56 |==================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 33.98 |=================================================================== b . 34.08 |==================================================================== c . 33.99 |=================================================================== d . 34.25 |==================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 10.36 |=================================================================== b . 10.37 |=================================================================== c . 10.35 |=================================================================== d . 10.45 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 475.13 |=================================================================== b . 474.37 |=================================================================== c . 475.35 |=================================================================== d . 475.45 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 31.58 |==================================================================== b . 31.73 |==================================================================== c . 31.56 |==================================================================== d . 30.29 |================================================================= NWChem 7.0.2 Input: C240 Buckyball Seconds < Lower Is Better a . 1744.0 |================================================================== b . 1730.7 |================================================================== c . 1757.3 |=================================================================== d . 1748.0 |=================================================================== WRF 4.2.2 Input: conus 2.5km Seconds < Lower Is Better a . 5566.73 |================================================================= b . 5600.98 |================================================================== c . 5583.11 |================================================================== d . 5617.20 |==================================================================