net INTEL XEON PLATINUM 8592+ testing with a Quanta Cloud S6Q-MB-MPS (3B05.TEL4P1 BIOS) and ASPEED on Ubuntu 23.10 via the Phoronix Test Suite. a: Processor: INTEL XEON PLATINUM 8592+ @ 3.90GHz (64 Cores / 128 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3B05.TEL4P1 BIOS), Chipset: Intel Device 1bce, Memory: 512GB, Disk: 3201GB Micron_7450_MTFDKCC3T2TFS + 0GB Virtual HDisk0 + 0GB Virtual HDisk2 + 0GB Virtual HDisk1 + 0GB Virtual HDisk3, Graphics: ASPEED OS: Ubuntu 23.10, Kernel: 6.6.0-rc5-phx-patched (x86_64), Desktop: GNOME Shell 45.0, Display Server: X Server 1.21.1.7, Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1200 b: Processor: INTEL XEON PLATINUM 8592+ @ 3.90GHz (64 Cores / 128 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3B05.TEL4P1 BIOS), Chipset: Intel Device 1bce, Memory: 512GB, Disk: 3201GB Micron_7450_MTFDKCC3T2TFS + 0GB Virtual HDisk0 + 0GB Virtual HDisk2 + 0GB Virtual HDisk1 + 0GB Virtual HDisk3, Graphics: ASPEED OS: Ubuntu 23.10, Kernel: 6.6.0-rc5-phx-patched (x86_64), Desktop: GNOME Shell 45.0, Display Server: X Server 1.21.1.7, Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1200 c: Processor: INTEL XEON PLATINUM 8592+ @ 3.90GHz (64 Cores / 128 Threads), Motherboard: Quanta Cloud S6Q-MB-MPS (3B05.TEL4P1 BIOS), Chipset: Intel Device 1bce, Memory: 512GB, Disk: 3201GB Micron_7450_MTFDKCC3T2TFS + 0GB Virtual HDisk0 + 0GB Virtual HDisk2 + 0GB Virtual HDisk1 + 0GB Virtual HDisk3, Graphics: ASPEED OS: Ubuntu 23.10, Kernel: 6.6.0-rc5-phx-patched (x86_64), Desktop: GNOME Shell 45.0, Display Server: X Server 1.21.1.7, Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1200 Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 28.46 |================================================================= b . 28.87 |================================================================== c . 29.68 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 35.10 |==================================================================== b . 34.62 |=================================================================== c . 33.72 |================================================================= SVT-AV1 1.8 Encoder Mode: Preset 8 - Input: Bosphorus 1080p Frames Per Second > Higher Is Better a . 139.81 |================================================================== b . 141.98 |=================================================================== c . 141.87 |=================================================================== LeelaChessZero 0.30 Backend: BLAS Nodes Per Second > Higher Is Better a . 67 |====================================================================== b . 67 |====================================================================== c . 68 |======================================================================= Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 5.3729 |=================================================================== b . 5.3005 |================================================================== c . 5.3092 |================================================================== SVT-AV1 1.8 Encoder Mode: Preset 4 - Input: Bosphorus 1080p Frames Per Second > Higher Is Better a . 20.77 |==================================================================== b . 20.59 |=================================================================== c . 20.49 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 186.03 |================================================================== b . 188.54 |=================================================================== c . 188.23 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 96.13 |=================================================================== b . 95.63 |=================================================================== c . 96.85 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 332.81 |=================================================================== b . 334.32 |=================================================================== c . 330.34 |================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 34.21 |==================================================================== b . 34.45 |==================================================================== c . 34.04 |=================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 934.65 |=================================================================== b . 928.45 |================================================================== c . 939.54 |=================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 3.4825 |=================================================================== b . 3.4417 |================================================================== c . 3.4767 |=================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 286.89 |================================================================== b . 290.28 |=================================================================== c . 287.35 |================================================================== SVT-AV1 1.8 Encoder Mode: Preset 8 - Input: Bosphorus 4K Frames Per Second > Higher Is Better a . 75.14 |=================================================================== b . 75.74 |==================================================================== c . 74.93 |=================================================================== Xmrig 6.21 Variant: KawPow - Hash Count: 1M H/s > Higher Is Better a . 39936.2 |================================================================= b . 40344.7 |================================================================== c . 40242.5 |================================================================== SVT-AV1 1.8 Encoder Mode: Preset 13 - Input: Bosphorus 4K Frames Per Second > Higher Is Better a . 227.29 |================================================================== b . 229.56 |=================================================================== c . 229.08 |=================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 67.87 |==================================================================== b . 68.18 |==================================================================== c . 67.58 |=================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 14.73 |==================================================================== b . 14.66 |=================================================================== c . 14.79 |==================================================================== Xmrig 6.21 Variant: CryptoNight-Femto UPX2 - Hash Count: 1M H/s > Higher Is Better a . 40425.9 |================================================================== b . 40080.4 |================================================================= c . 40331.1 |================================================================== Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 34.28 |==================================================================== b . 34.22 |==================================================================== c . 34.01 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 933.01 |================================================================== b . 934.52 |=================================================================== c . 940.22 |=================================================================== SVT-AV1 1.8 Encoder Mode: Preset 13 - Input: Bosphorus 1080p Frames Per Second > Higher Is Better a . 632.46 |================================================================== b . 636.91 |=================================================================== c . 637.34 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 29.00 |==================================================================== b . 29.00 |==================================================================== c . 29.20 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 34.47 |==================================================================== b . 34.48 |==================================================================== c . 34.24 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 210.96 |=================================================================== b . 210.81 |=================================================================== c . 209.61 |=================================================================== Xmrig 6.21 Variant: Monero - Hash Count: 1M H/s > Higher Is Better a . 40329.6 |================================================================== b . 40153.2 |================================================================== c . 40072.2 |================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 4.7354 |=================================================================== b . 4.7386 |=================================================================== c . 4.7654 |=================================================================== Xmrig 6.21 Variant: GhostRider - Hash Count: 1M H/s > Higher Is Better a . 6862.9 |=================================================================== b . 6857.8 |=================================================================== c . 6820.8 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 164.94 |=================================================================== b . 164.90 |=================================================================== c . 163.97 |=================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 701.79 |=================================================================== b . 705.83 |=================================================================== c . 701.71 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 49.90 |==================================================================== b . 50.17 |==================================================================== c . 50.19 |==================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 1.4200 |=================================================================== b . 1.4123 |=================================================================== c . 1.4205 |=================================================================== SVT-AV1 1.8 Encoder Mode: Preset 12 - Input: Bosphorus 1080p Frames Per Second > Higher Is Better a . 545.91 |=================================================================== b . 543.73 |=================================================================== c . 546.88 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 641.10 |=================================================================== b . 637.61 |=================================================================== c . 637.42 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 6.0577 |=================================================================== b . 6.0589 |=================================================================== c . 6.0922 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 28.89 |==================================================================== b . 28.88 |==================================================================== c . 29.02 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 34.61 |==================================================================== b . 34.63 |==================================================================== c . 34.45 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 290.01 |=================================================================== b . 288.68 |=================================================================== c . 289.81 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 3.4452 |=================================================================== b . 3.4609 |=================================================================== c . 3.4473 |=================================================================== SVT-AV1 1.8 Encoder Mode: Preset 12 - Input: Bosphorus 4K Frames Per Second > Higher Is Better a . 228.87 |=================================================================== b . 229.51 |=================================================================== c . 228.52 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 71.99 |==================================================================== b . 71.68 |==================================================================== c . 71.97 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 72.05 |==================================================================== b . 72.09 |==================================================================== c . 71.78 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 442.22 |=================================================================== b . 441.48 |=================================================================== c . 443.16 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 452.20 |=================================================================== b . 452.16 |=================================================================== c . 450.49 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 70.74 |==================================================================== b . 70.74 |==================================================================== c . 71.00 |==================================================================== Xmrig 6.21 Variant: CryptoNight-Heavy - Hash Count: 1M H/s > Higher Is Better a . 40394.9 |================================================================== b . 40282.2 |================================================================== c . 40286.6 |================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 31.22 |==================================================================== b . 31.15 |==================================================================== c . 31.19 |==================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 32.02 |==================================================================== b . 32.10 |==================================================================== c . 32.06 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 442.31 |=================================================================== b . 443.40 |=================================================================== c . 442.75 |=================================================================== SVT-AV1 1.8 Encoder Mode: Preset 4 - Input: Bosphorus 4K Frames Per Second > Higher Is Better a . 7.156 |==================================================================== b . 7.168 |==================================================================== c . 7.151 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 436.52 |=================================================================== b . 436.33 |=================================================================== c . 435.49 |=================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 5706.70 |================================================================== b . 5717.52 |================================================================== c . 5704.95 |================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 73.27 |==================================================================== b . 73.29 |==================================================================== c . 73.43 |==================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 5.5953 |=================================================================== b . 5.5844 |=================================================================== c . 5.5965 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 166.66 |=================================================================== b . 166.51 |=================================================================== c . 166.31 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 5.9970 |=================================================================== b . 6.0023 |=================================================================== c . 6.0096 |=================================================================== Xmrig 6.21 Variant: Wownero - Hash Count: 1M H/s > Higher Is Better a . 43088.3 |================================================================== b . 42998.7 |================================================================== c . 43080.4 |================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 370.73 |=================================================================== b . 370.78 |=================================================================== c . 371.30 |=================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 86.30 |==================================================================== b . 86.29 |==================================================================== c . 86.17 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 13.75 |==================================================================== b . 13.75 |==================================================================== c . 13.76 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 2323.98 |================================================================== b . 2324.74 |================================================================== c . 2322.58 |================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 34.07 |==================================================================== b . 34.09 |==================================================================== c . 34.08 |==================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 938.07 |=================================================================== b . 937.58 |=================================================================== c . 937.86 |=================================================================== LeelaChessZero 0.30 Backend: Eigen Nodes Per Second > Higher Is Better a . 796 |==================================================================== b . 805 |===================================================================== c . 817 |======================================================================