ddfg 2 x AMD EPYC 9684X 96-Core testing with a AMD Titanite_4G (RTI1007B BIOS) and ASPEED on Ubuntu 23.10 via the Phoronix Test Suite. a: Processor: 2 x AMD EPYC 9684X 96-Core @ 2.55GHz (192 Cores / 384 Threads), Motherboard: AMD Titanite_4G (RTI1007B BIOS), Chipset: AMD Device 14a4, Memory: 1520GB, Disk: 3201GB Micron_7450_MTFDKCB3T2TFS + 257GB Flash Drive, Graphics: ASPEED, Network: Broadcom NetXtreme BCM5720 PCIe OS: Ubuntu 23.10, Kernel: 6.5.0-25-generic (x86_64), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 640x480 b: Processor: 2 x AMD EPYC 9684X 96-Core @ 2.55GHz (192 Cores / 384 Threads), Motherboard: AMD Titanite_4G (RTI1007B BIOS), Chipset: AMD Device 14a4, Memory: 1520GB, Disk: 3201GB Micron_7450_MTFDKCB3T2TFS + 257GB Flash Drive, Graphics: ASPEED, Network: Broadcom NetXtreme BCM5720 PCIe OS: Ubuntu 23.10, Kernel: 6.5.0-25-generic (x86_64), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 640x480 c: Processor: 2 x AMD EPYC 9684X 96-Core @ 2.55GHz (192 Cores / 384 Threads), Motherboard: AMD Titanite_4G (RTI1007B BIOS), Chipset: AMD Device 14a4, Memory: 1520GB, Disk: 3201GB Micron_7450_MTFDKCB3T2TFS + 257GB Flash Drive, Graphics: ASPEED, Network: Broadcom NetXtreme BCM5720 PCIe OS: Ubuntu 23.10, Kernel: 6.5.0-25-generic (x86_64), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 640x480 d: Processor: 2 x AMD EPYC 9684X 96-Core @ 2.55GHz (192 Cores / 384 Threads), Motherboard: AMD Titanite_4G (RTI1007B BIOS), Chipset: AMD Device 14a4, Memory: 1520GB, Disk: 3201GB Micron_7450_MTFDKCB3T2TFS + 257GB Flash Drive, Graphics: ASPEED, Network: Broadcom NetXtreme BCM5720 PCIe OS: Ubuntu 23.10, Kernel: 6.5.0-25-generic (x86_64), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 640x480 SVT-AV1 2.0 Encoder Mode: Preset 4 - Input: Bosphorus 4K Frames Per Second > Higher Is Better a . 8.557 |=================================================================== b . 8.575 |==================================================================== c . 8.484 |=================================================================== d . 8.634 |==================================================================== SVT-AV1 2.0 Encoder Mode: Preset 8 - Input: Bosphorus 4K Frames Per Second > Higher Is Better a . 89.28 |=================================================================== b . 91.29 |==================================================================== c . 90.44 |=================================================================== d . 86.59 |================================================================ SVT-AV1 2.0 Encoder Mode: Preset 12 - Input: Bosphorus 4K Frames Per Second > Higher Is Better a . 149.07 |============================================================= b . 163.18 |=================================================================== c . 160.56 |================================================================== d . 163.50 |=================================================================== SVT-AV1 2.0 Encoder Mode: Preset 13 - Input: Bosphorus 4K Frames Per Second > Higher Is Better a . 159.22 |================================================================== b . 161.65 |=================================================================== c . 162.02 |=================================================================== d . 162.38 |=================================================================== SVT-AV1 2.0 Encoder Mode: Preset 4 - Input: Bosphorus 1080p Frames Per Second > Higher Is Better a . 22.96 |=================================================================== b . 22.90 |================================================================== c . 23.47 |==================================================================== d . 22.71 |================================================================== SVT-AV1 2.0 Encoder Mode: Preset 8 - Input: Bosphorus 1080p Frames Per Second > Higher Is Better a . 176.67 |================================================================ b . 182.53 |================================================================== c . 184.13 |=================================================================== d . 183.29 |=================================================================== SVT-AV1 2.0 Encoder Mode: Preset 12 - Input: Bosphorus 1080p Frames Per Second > Higher Is Better a . 572.35 |================================================================== b . 566.99 |================================================================== c . 568.21 |================================================================== d . 577.44 |=================================================================== SVT-AV1 2.0 Encoder Mode: Preset 13 - Input: Bosphorus 1080p Frames Per Second > Higher Is Better a . 564.41 |============================================================== b . 606.38 |=================================================================== c . 588.45 |================================================================= d . 598.04 |================================================================== Primesieve 12.1 Length: 1e12 Seconds < Lower Is Better a . 1.164 |==================================================================== b . 1.164 |==================================================================== c . 1.156 |==================================================================== d . 1.142 |=================================================================== Primesieve 12.1 Length: 1e13 Seconds < Lower Is Better a . 11.79 |==================================================================== b . 11.85 |==================================================================== c . 11.87 |==================================================================== d . 11.84 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 132.95 |=================================================================== b . 132.98 |=================================================================== c . 132.88 |=================================================================== d . 133.01 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 713.54 |=================================================================== b . 717.49 |=================================================================== c . 715.88 |=================================================================== d . 714.46 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 48.52 |==================================================================== b . 48.36 |==================================================================== c . 48.52 |==================================================================== d . 48.51 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 20.60 |==================================================================== b . 20.67 |==================================================================== c . 20.60 |==================================================================== d . 20.61 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 5566.39 |================================================================== b . 5538.98 |================================================================== c . 5534.26 |================================================================== d . 5554.03 |================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 17.22 |==================================================================== b . 17.30 |==================================================================== c . 17.32 |==================================================================== d . 17.26 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 191.51 |================================================================== b . 191.77 |================================================================== c . 192.23 |================================================================== d . 194.47 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 5.2180 |=================================================================== b . 5.2109 |=================================================================== c . 5.1985 |=================================================================== d . 5.1383 |================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 1762.53 |================================================================== b . 1758.81 |================================================================== c . 1759.96 |================================================================== d . 1763.42 |================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 54.37 |==================================================================== b . 54.49 |==================================================================== c . 54.46 |==================================================================== d . 54.36 |==================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 206.82 |================================================================== b . 209.77 |=================================================================== c . 208.74 |=================================================================== d . 208.98 |=================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 4.8321 |=================================================================== b . 4.7642 |================================================================== c . 4.7875 |================================================================== d . 4.7823 |================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 17421.47 |================================================================= b . 17350.46 |================================================================= c . 17333.35 |================================================================= d . 17400.58 |================================================================= Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 5.4946 |=================================================================== b . 5.5170 |=================================================================== c . 5.5224 |=================================================================== d . 5.5020 |=================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 807.17 |=================================================================== b . 804.40 |=================================================================== c . 806.23 |=================================================================== d . 806.32 |=================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 1.2364 |=================================================================== b . 1.2409 |=================================================================== c . 1.2381 |=================================================================== d . 1.2375 |=================================================================== Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 2.9358 |=================================================================== b . 2.8607 |================================================================= c . 2.5008 |========================================================= d . 2.9271 |=================================================================== Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 24242.38 |====================================================== b . 24888.16 |======================================================== c . 28968.95 |================================================================= d . 24331.06 |======================================================= Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 20.83 |==================================================================== b . 20.81 |==================================================================== c . 20.73 |==================================================================== d . 20.46 |=================================================================== Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 47.97 |=================================================================== b . 48.03 |=================================================================== c . 48.22 |=================================================================== d . 48.84 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 1763.19 |================================================================== b . 1756.17 |================================================================== c . 1756.56 |================================================================== d . 1758.31 |================================================================== Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 54.35 |==================================================================== b . 54.57 |==================================================================== c . 54.56 |==================================================================== d . 54.51 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 207.67 |=================================================================== b . 208.72 |=================================================================== c . 208.63 |=================================================================== d . 209.19 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 4.8123 |=================================================================== b . 4.7882 |=================================================================== c . 4.7905 |=================================================================== d . 4.7773 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 798.62 |=================================================================== b . 796.60 |=================================================================== c . 795.03 |=================================================================== d . 797.71 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 119.91 |=================================================================== b . 120.18 |=================================================================== c . 120.40 |=================================================================== d . 120.00 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 212.18 |=================================================================== b . 212.49 |=================================================================== c . 212.64 |=================================================================== d . 213.56 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 4.7104 |=================================================================== b . 4.7035 |=================================================================== c . 4.7002 |=================================================================== d . 4.6801 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 1144.50 |================================================================== b . 1142.99 |================================================================== c . 1141.24 |================================================================== d . 1144.01 |================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 83.66 |==================================================================== b . 83.85 |==================================================================== c . 83.93 |==================================================================== d . 83.77 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 225.47 |=================================================================== b . 225.45 |=================================================================== c . 225.44 |=================================================================== d . 225.56 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 4.4325 |=================================================================== b . 4.4329 |=================================================================== c . 4.4329 |=================================================================== d . 4.4307 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 249.72 |=================================================================== b . 249.40 |=================================================================== c . 249.09 |=================================================================== d . 249.78 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 381.04 |=================================================================== b . 382.12 |=================================================================== c . 382.17 |=================================================================== d . 381.12 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 64.89 |==================================================================== b . 64.86 |==================================================================== c . 64.92 |==================================================================== d . 65.00 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 15.39 |==================================================================== b . 15.40 |==================================================================== c . 15.38 |==================================================================== d . 15.37 |==================================================================== Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 2598.21 |================================================================== b . 2595.18 |================================================================== c . 2595.48 |================================================================== d . 2592.85 |================================================================== Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 36.89 |==================================================================== b . 36.93 |==================================================================== c . 36.92 |==================================================================== d . 36.96 |==================================================================== Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 68.58 |==================================================================== b . 68.40 |==================================================================== c . 68.56 |==================================================================== d . 68.77 |==================================================================== Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 14.57 |==================================================================== b . 14.61 |==================================================================== c . 14.58 |==================================================================== d . 14.53 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 132.95 |=================================================================== b . 132.80 |================================================================== c . 133.81 |=================================================================== d . 132.91 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 715.46 |=================================================================== b . 717.26 |=================================================================== c . 709.26 |================================================================== d . 715.43 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 48.53 |==================================================================== b . 48.42 |==================================================================== c . 48.38 |==================================================================== d . 48.38 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 20.60 |==================================================================== b . 20.64 |==================================================================== c . 20.66 |==================================================================== d . 20.66 |====================================================================