8380 2023 2 x Intel Xeon Platinum 8380 testing with a Intel M50CYP2SB2U (SE5C6200.86B.0022.D08.2103221623 BIOS) and ASPEED on Ubuntu 22.10 via the Phoronix Test Suite. a: Processor: 2 x Intel Xeon Platinum 8380 @ 3.40GHz (80 Cores / 160 Threads), Motherboard: Intel M50CYP2SB2U (SE5C6200.86B.0022.D08.2103221623 BIOS), Chipset: Intel Ice Lake IEH, Memory: 512GB, Disk: 7682GB INTEL SSDPF2KX076TZ, Graphics: ASPEED, Monitor: VE228, Network: 2 x Intel X710 for 10GBASE-T + 2 x Intel E810-C for QSFP OS: Ubuntu 22.10, Kernel: 6.2.0-rc5-phx-dodt (x86_64), Desktop: GNOME Shell, Display Server: X Server 1.21.1.3, Vulkan: 1.3.211, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1080 b: Processor: 2 x Intel Xeon Platinum 8380 @ 3.40GHz (80 Cores / 160 Threads), Motherboard: Intel M50CYP2SB2U (SE5C6200.86B.0022.D08.2103221623 BIOS), Chipset: Intel Ice Lake IEH, Memory: 512GB, Disk: 7682GB INTEL SSDPF2KX076TZ, Graphics: ASPEED, Monitor: VE228, Network: 2 x Intel X710 for 10GBASE-T + 2 x Intel E810-C for QSFP OS: Ubuntu 22.10, Kernel: 6.2.0-rc5-phx-dodt (x86_64), Desktop: GNOME Shell, Display Server: X Server 1.21.1.3, Vulkan: 1.3.211, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1080 c: Processor: 2 x Intel Xeon Platinum 8380 @ 3.40GHz (80 Cores / 160 Threads), Motherboard: Intel M50CYP2SB2U (SE5C6200.86B.0022.D08.2103221623 BIOS), Chipset: Intel Ice Lake IEH, Memory: 512GB, Disk: 7682GB INTEL SSDPF2KX076TZ, Graphics: ASPEED, Monitor: VE228, Network: 2 x Intel X710 for 10GBASE-T + 2 x Intel E810-C for QSFP OS: Ubuntu 22.10, Kernel: 6.2.0-rc5-phx-dodt (x86_64), Desktop: GNOME Shell, Display Server: X Server 1.21.1.3, Vulkan: 1.3.224, Compiler: GCC 12.2.0, File-System: ext4, Screen Resolution: 1920x1080 Neural Magic DeepSparse 1.3.2 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 13.65 |=================================================================== b . 12.77 |=============================================================== c . 13.82 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 73.18 |================================================================ b . 78.28 |==================================================================== c . 72.33 |=============================================================== uvg266 0.4.1 Video Input: Bosphorus 4K - Video Preset: Super Fast Frames Per Second > Higher Is Better a . 41.47 |================================================================ b . 44.36 |==================================================================== c . 42.29 |================================================================= uvg266 0.4.1 Video Input: Bosphorus 4K - Video Preset: Ultra Fast Frames Per Second > Higher Is Better a . 45.37 |==================================================================== b . 42.43 |================================================================ c . 44.99 |=================================================================== Kvazaar 2.2 Video Input: Bosphorus 4K - Video Preset: Ultra Fast Frames Per Second > Higher Is Better a . 49.15 |=================================================================== b . 49.79 |==================================================================== c . 46.99 |================================================================ uvg266 0.4.1 Video Input: Bosphorus 4K - Video Preset: Very Fast Frames Per Second > Higher Is Better a . 41.18 |================================================================== b . 42.61 |==================================================================== c . 40.49 |================================================================= uvg266 0.4.1 Video Input: Bosphorus 1080p - Video Preset: Super Fast Frames Per Second > Higher Is Better a . 144.87 |================================================================= b . 149.29 |=================================================================== c . 143.45 |================================================================ Kvazaar 2.2 Video Input: Bosphorus 1080p - Video Preset: Super Fast Frames Per Second > Higher Is Better a . 189.75 |=================================================================== b . 190.56 |=================================================================== c . 183.65 |================================================================= RocksDB 7.9.2 Test: Random Fill Sync Op/s > Higher Is Better a . 508007 |=================================================================== b . 490210 |================================================================= c . 508639 |=================================================================== RocksDB 7.9.2 Test: Read While Writing Op/s > Higher Is Better a . 7701319 |================================================================ b . 7975258 |================================================================== c . 7764276 |================================================================ uvg266 0.4.1 Video Input: Bosphorus 1080p - Video Preset: Ultra Fast Frames Per Second > Higher Is Better a . 145.42 |================================================================= b . 146.00 |================================================================= c . 150.50 |=================================================================== Kvazaar 2.2 Video Input: Bosphorus 1080p - Video Preset: Very Fast Frames Per Second > Higher Is Better a . 185.67 |=================================================================== b . 179.78 |================================================================= c . 181.91 |================================================================== ClickHouse 22.12.3.5 100M Rows Hits Dataset, First Run / Cold Cache Queries Per Minute, Geo Mean > Higher Is Better a . 399.61 |=================================================================== b . 398.98 |=================================================================== c . 387.30 |================================================================= Neural Magic DeepSparse 1.3.2 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 222.84 |=================================================================== b . 216.63 |================================================================= c . 218.10 |================================================================== Kvazaar 2.2 Video Input: Bosphorus 1080p - Video Preset: Medium Frames Per Second > Higher Is Better a . 88.85 |==================================================================== b . 86.38 |================================================================== c . 86.96 |=================================================================== Kvazaar 2.2 Video Input: Bosphorus 1080p - Video Preset: Ultra Fast Frames Per Second > Higher Is Better a . 189.02 |=================================================================== b . 183.89 |================================================================= c . 184.65 |================================================================= RocksDB 7.9.2 Test: Random Read Op/s > Higher Is Better a . 283388484 |================================================================ b . 278662255 |=============================================================== c . 275851418 |============================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Question Answering, BERT base uncased SQuaD 12layer Pruned90 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 179.33 |================================================================= b . 184.17 |=================================================================== c . 182.88 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 221.09 |================================================================= b . 220.48 |================================================================= c . 226.39 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 180.85 |=================================================================== b . 180.99 |=================================================================== c . 176.31 |================================================================= RocksDB 7.9.2 Test: Random Fill Op/s > Higher Is Better a . 608380 |================================================================= b . 609813 |================================================================= c . 624361 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 52.99 |=================================================================== b . 53.58 |==================================================================== c . 52.21 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, BERT base uncased SST2 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 18.86 |=================================================================== b . 18.66 |================================================================== c . 19.15 |==================================================================== uvg266 0.4.1 Video Input: Bosphorus 1080p - Video Preset: Very Fast Frames Per Second > Higher Is Better a . 144.66 |================================================================== b . 146.27 |=================================================================== c . 142.73 |================================================================= Kvazaar 2.2 Video Input: Bosphorus 1080p - Video Preset: Slow Frames Per Second > Higher Is Better a . 84.78 |=================================================================== b . 86.02 |==================================================================== c . 83.98 |================================================================== ClickHouse 22.12.3.5 100M Rows Hits Dataset, Third Run Queries Per Minute, Geo Mean > Higher Is Better a . 427.97 |=================================================================== b . 418.34 |================================================================= c . 426.64 |=================================================================== Kvazaar 2.2 Video Input: Bosphorus 4K - Video Preset: Very Fast Frames Per Second > Higher Is Better a . 44.49 |==================================================================== b . 44.30 |==================================================================== c . 43.49 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better b . 896.69 |=================================================================== c . 878.86 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better b . 44.58 |=================================================================== c . 45.45 |==================================================================== Kvazaar 2.2 Video Input: Bosphorus 4K - Video Preset: Medium Frames Per Second > Higher Is Better a . 20.28 |=================================================================== b . 20.46 |==================================================================== c . 20.17 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 20.27 |==================================================================== b . 20.31 |==================================================================== c . 20.04 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 49.32 |=================================================================== b . 49.24 |=================================================================== c . 49.89 |==================================================================== RocksDB 7.9.2 Test: Read Random Write Random Op/s > Higher Is Better a . 2711634 |================================================================== b . 2676508 |================================================================= c . 2711331 |================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 82.24 |==================================================================== b . 82.67 |==================================================================== c . 81.68 |=================================================================== RocksDB 7.9.2 Test: Update Random Op/s > Higher Is Better a . 594126 |=================================================================== b . 587046 |================================================================== c . 588321 |================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 833.90 |=================================================================== b . 824.22 |================================================================== c . 828.26 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 48.74 |==================================================================== b . 48.34 |=================================================================== c . 48.20 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 47.94 |=================================================================== b . 48.45 |==================================================================== c . 48.22 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 485.26 |=================================================================== b . 482.34 |================================================================== c . 487.48 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 318.73 |=================================================================== b . 315.40 |================================================================== c . 316.61 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 125.19 |================================================================== b . 126.46 |=================================================================== c . 125.97 |=================================================================== uvg266 0.4.1 Video Input: Bosphorus 1080p - Video Preset: Medium Frames Per Second > Higher Is Better a . 55.14 |==================================================================== b . 55.51 |==================================================================== c . 55.00 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 49.52 |=================================================================== b . 49.97 |==================================================================== c . 49.61 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 20.19 |==================================================================== b . 20.01 |=================================================================== c . 20.16 |==================================================================== ClickHouse 22.12.3.5 100M Rows Hits Dataset, Second Run Queries Per Minute, Geo Mean > Higher Is Better a . 424.16 |=================================================================== b . 421.19 |=================================================================== c . 420.61 |================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 175.35 |=================================================================== b . 175.92 |=================================================================== c . 174.65 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Sentiment Analysis, 80% Pruned Quantized BERT Base Uncased - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 5.6930 |=================================================================== b . 5.6761 |=================================================================== c . 5.7175 |=================================================================== Kvazaar 2.2 Video Input: Bosphorus 4K - Video Preset: Super Fast Frames Per Second > Higher Is Better a . 47.91 |==================================================================== b . 48.11 |==================================================================== c . 48.23 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 817.15 |=================================================================== b . 822.42 |=================================================================== c . 820.75 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 10.05 |==================================================================== b . 10.11 |==================================================================== c . 10.11 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 99.41 |==================================================================== b . 98.80 |==================================================================== c . 98.79 |==================================================================== uvg266 0.4.1 Video Input: Bosphorus 4K - Video Preset: Slow Frames Per Second > Higher Is Better a . 14.53 |==================================================================== b . 14.49 |==================================================================== c . 14.44 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 452.76 |=================================================================== b . 451.84 |=================================================================== c . 454.25 |=================================================================== RocksDB 7.9.2 Test: Sequential Fill Op/s > Higher Is Better a . 615043 |=================================================================== b . 615815 |=================================================================== c . 618232 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 88.20 |==================================================================== b . 88.39 |==================================================================== c . 87.95 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 223.31 |=================================================================== b . 222.29 |=================================================================== c . 223.40 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 4.4681 |=================================================================== b . 4.4899 |=================================================================== c . 4.4680 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 34.34 |==================================================================== b . 34.41 |==================================================================== c . 34.24 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 29.10 |==================================================================== b . 29.04 |==================================================================== c . 29.18 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 83.36 |==================================================================== b . 83.00 |==================================================================== c . 83.31 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 11.99 |==================================================================== b . 12.04 |==================================================================== c . 11.99 |==================================================================== uvg266 0.4.1 Video Input: Bosphorus 1080p - Video Preset: Slow Frames Per Second > Higher Is Better a . 49.65 |==================================================================== b . 49.85 |==================================================================== c . 49.79 |==================================================================== uvg266 0.4.1 Video Input: Bosphorus 4K - Video Preset: Medium Frames Per Second > Higher Is Better a . 16.29 |==================================================================== b . 16.25 |==================================================================== c . 16.30 |==================================================================== Kvazaar 2.2 Video Input: Bosphorus 4K - Video Preset: Slow Frames Per Second > Higher Is Better a . 19.81 |==================================================================== b . 19.79 |==================================================================== c . 19.75 |==================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 820.54 |=================================================================== b . 821.79 |=================================================================== c . 820.37 |=================================================================== Neural Magic DeepSparse 1.3.2 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 48.24 |==================================================================== b . 48.25 |==================================================================== c . 48.30 |====================================================================