rocksdb deepsparse epyc 2 x AMD EPYC 9654 96-Core testing with a AMD Titanite_4G (RTI1002E BIOS) and ASPEED on CentOS Stream 9 via the Phoronix Test Suite. a: Processor: 2 x AMD EPYC 9654 96-Core @ 2.40GHz (192 Cores / 384 Threads), Motherboard: AMD Titanite_4G (RTI1002E BIOS), Chipset: AMD Device 14a4, Memory: 1520GB, Disk: 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 257GB Flash Drive, Graphics: ASPEED, Monitor: VGA HDMI, Network: Broadcom NetXtreme BCM5720 PCIe OS: CentOS Stream 9, Kernel: 5.14.0-239.el9.x86_64 (x86_64), Desktop: GNOME Shell 40.10, Display Server: X Server, Compiler: GCC 11.3.1 20221121, File-System: xfs, Screen Resolution: 1024x768 b: Processor: 2 x AMD EPYC 9654 96-Core @ 2.40GHz (192 Cores / 384 Threads), Motherboard: AMD Titanite_4G (RTI1002E BIOS), Chipset: AMD Device 14a4, Memory: 1520GB, Disk: 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 257GB Flash Drive, Graphics: ASPEED, Monitor: VGA HDMI, Network: Broadcom NetXtreme BCM5720 PCIe OS: CentOS Stream 9, Kernel: 5.14.0-239.el9.x86_64 (x86_64), Desktop: GNOME Shell 40.10, Display Server: X Server, Compiler: GCC 11.3.1 20221121, File-System: xfs, Screen Resolution: 1024x768 c: Processor: 2 x AMD EPYC 9654 96-Core @ 2.40GHz (192 Cores / 384 Threads), Motherboard: AMD Titanite_4G (RTI1002E BIOS), Chipset: AMD Device 14a4, Memory: 1520GB, Disk: 2 x 1920GB SAMSUNG MZWLJ1T9HBJR-00007 + 257GB Flash Drive, Graphics: ASPEED, Monitor: VGA HDMI, Network: Broadcom NetXtreme BCM5720 PCIe OS: CentOS Stream 9, Kernel: 5.14.0-239.el9.x86_64 (x86_64), Desktop: GNOME Shell 40.10, Display Server: X Server, Compiler: GCC 11.3.1 20221121, File-System: xfs, Screen Resolution: 1024x768 Neural Magic DeepSparse 1.3.2 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 84.97 |==================================================================== b . 85.26 |==================================================================== c . 85.17 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 1128.13 |================================================================== b . 1125.11 |================================================================== c . 1125.58 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 33.87 |==================================================================== b . 33.67 |==================================================================== c . 33.78 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 29.52 |==================================================================== b . 29.69 |==================================================================== c . 29.60 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 2846.15 |================================================================== b . 2825.65 |================================================================== c . 2835.42 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 33.66 |==================================================================== b . 33.90 |==================================================================== c . 33.79 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 192.16 |================================================================== b . 194.99 |=================================================================== c . 195.55 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 5.2014 |=================================================================== b . 5.1258 |================================================================== c . 5.1115 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 758.78 |=================================================================== b . 759.40 |=================================================================== c . 750.30 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 126.16 |================================================================== b . 126.14 |================================================================== c . 127.65 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 97.09 |==================================================================== b . 96.67 |==================================================================== c . 96.93 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 10.29 |==================================================================== b . 10.34 |==================================================================== c . 10.31 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 808.68 |=================================================================== b . 809.59 |=================================================================== c . 808.00 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 118.46 |=================================================================== b . 118.31 |=================================================================== c . 118.51 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 104.46 |===================================================== b . 131.83 |=================================================================== c . 120.18 |============================================================= Neural Magic DeepSparse 1.3.2 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 9.5625 |=================================================================== b . 7.5725 |===================================================== c . 8.3091 |========================================================== Neural Magic DeepSparse 1.3.2 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 1975.94 |================================================================== b . 1972.09 |================================================================== c . 1965.81 |================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 48.49 |==================================================================== b . 48.58 |==================================================================== c . 48.75 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 185.86 |=================================================================== b . 179.77 |================================================================= c . 179.08 |================================================================= Neural Magic DeepSparse 1.3.2 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 5.3776 |================================================================= b . 5.5603 |=================================================================== c . 5.5814 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 1204.63 |================================================================== b . 1201.02 |================================================================== c . 1207.64 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 79.59 |==================================================================== b . 79.77 |==================================================================== c . 79.27 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 189.94 |================================================================== b . 191.52 |=================================================================== c . 192.01 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 5.2599 |=================================================================== b . 5.2171 |================================================================== c . 5.2031 |================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 282.53 |=================================================================== b . 283.31 |=================================================================== c . 282.44 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 337.71 |=================================================================== b . 337.05 |=================================================================== c . 337.85 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 59.89 |==================================================================== b . 59.78 |==================================================================== c . 59.44 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 16.68 |=================================================================== b . 16.71 |==================================================================== c . 16.81 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 619.52 |=================================================================== b . 615.67 |================================================================== c . 621.39 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 154.50 |=================================================================== b . 155.46 |=================================================================== c . 153.98 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 86.96 |==================================================================== b . 87.00 |==================================================================== c . 86.92 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 11.49 |==================================================================== b . 11.49 |==================================================================== c . 11.50 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 85.21 |==================================================================== b . 85.29 |==================================================================== c . 85.20 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 1124.94 |================================================================== b . 1123.83 |================================================================== c . 1123.43 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 34.00 |==================================================================== b . 33.58 |=================================================================== c . 33.70 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 29.41 |=================================================================== b . 29.77 |==================================================================== c . 29.66 |==================================================================== RocksDB 7.9.2 Test: Random Fill Op/s > Higher Is Better a . 446527 |=============================================================== b . 474262 |=================================================================== c . 465668 |================================================================== RocksDB 7.9.2 Test: Random Read Op/s > Higher Is Better a . 955753685 |================================================================ b . 953376399 |================================================================ c . 956838845 |================================================================ RocksDB 7.9.2 Test: Update Random Op/s > Higher Is Better a . 443071 |=================================================================== b . 433807 |================================================================== c . 433706 |================================================================== RocksDB 7.9.2 Test: Sequential Fill Op/s > Higher Is Better a . 449584 |=================================================================== b . 447613 |=================================================================== c . 448033 |=================================================================== RocksDB 7.9.2 Test: Random Fill Sync Op/s > Higher Is Better a . 384464 |=================================================================== b . 369406 |================================================================ c . 375291 |================================================================= RocksDB 7.9.2 Test: Read While Writing Op/s > Higher Is Better a . 15556265 |===================================================== b . 15596830 |====================================================== c . 18949029 |================================================================= RocksDB 7.9.2 Test: Read Random Write Random Op/s > Higher Is Better a . 1674556 |================================================================= b . 1694573 |================================================================== c . 1695311 |==================================================================