svt deepsparse AMD Ryzen Threadripper 7980X 64-Cores testing with a System76 Thelio Major (FA Z5 BIOS) and AMD Radeon Pro W7900 45GB on Ubuntu 23.10 via the Phoronix Test Suite. a: Processor: AMD Ryzen Threadripper 7980X 64-Cores @ 7.79GHz (64 Cores / 128 Threads), Motherboard: System76 Thelio Major (FA Z5 BIOS), Chipset: AMD Device 14a4, Memory: 4 x 32GB DRAM-4800MT/s Micron MTC20F1045S1RC48BA2, Disk: 1000GB CT1000T700SSD5, Graphics: AMD Radeon Pro W7900 45GB (1760/1124MHz), Audio: AMD Device 14cc, Monitor: DELL P2415Q, Network: Aquantia AQC113C NBase-T/IEEE + Realtek RTL8125 2.5GbE + Intel Wi-Fi 6 AX210/AX211/AX411 OS: Ubuntu 23.10, Kernel: 6.5.0-25-generic (x86_64), Desktop: GNOME Shell 45.2, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 23.2.1-1ubuntu3.1 (LLVM 15.0.7 DRM 3.54), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080 b: Processor: AMD Ryzen Threadripper 7980X 64-Cores @ 7.79GHz (64 Cores / 128 Threads), Motherboard: System76 Thelio Major (FA Z5 BIOS), Chipset: AMD Device 14a4, Memory: 4 x 32GB DRAM-4800MT/s Micron MTC20F1045S1RC48BA2, Disk: 1000GB CT1000T700SSD5, Graphics: AMD Radeon Pro W7900 45GB (1760/1124MHz), Audio: AMD Device 14cc, Monitor: DELL P2415Q, Network: Aquantia AQC113C NBase-T/IEEE + Realtek RTL8125 2.5GbE + Intel Wi-Fi 6 AX210/AX211/AX411 OS: Ubuntu 23.10, Kernel: 6.5.0-25-generic (x86_64), Desktop: GNOME Shell 45.2, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 23.2.1-1ubuntu3.1 (LLVM 15.0.7 DRM 3.54), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080 c: Processor: AMD Ryzen Threadripper 7980X 64-Cores @ 7.79GHz (64 Cores / 128 Threads), Motherboard: System76 Thelio Major (FA Z5 BIOS), Chipset: AMD Device 14a4, Memory: 4 x 32GB DRAM-4800MT/s Micron MTC20F1045S1RC48BA2, Disk: 1000GB CT1000T700SSD5, Graphics: AMD Radeon Pro W7900 45GB (1760/1124MHz), Audio: AMD Device 14cc, Monitor: DELL P2415Q, Network: Aquantia AQC113C NBase-T/IEEE + Realtek RTL8125 2.5GbE + Intel Wi-Fi 6 AX210/AX211/AX411 OS: Ubuntu 23.10, Kernel: 6.5.0-25-generic (x86_64), Desktop: GNOME Shell 45.2, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 23.2.1-1ubuntu3.1 (LLVM 15.0.7 DRM 3.54), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080 d: Processor: AMD Ryzen Threadripper 7980X 64-Cores @ 7.79GHz (64 Cores / 128 Threads), Motherboard: System76 Thelio Major (FA Z5 BIOS), Chipset: AMD Device 14a4, Memory: 4 x 32GB DRAM-4800MT/s Micron MTC20F1045S1RC48BA2, Disk: 1000GB CT1000T700SSD5, Graphics: AMD Radeon Pro W7900 45GB (1760/1124MHz), Audio: AMD Device 14cc, Monitor: DELL P2415Q, Network: Aquantia AQC113C NBase-T/IEEE + Realtek RTL8125 2.5GbE + Intel Wi-Fi 6 AX210/AX211/AX411 OS: Ubuntu 23.10, Kernel: 6.5.0-25-generic (x86_64), Desktop: GNOME Shell 45.2, Display Server: X Server + Wayland, OpenGL: 4.6 Mesa 23.2.1-1ubuntu3.1 (LLVM 15.0.7 DRM 3.54), Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080 SVT-AV1 2.0 Encoder Mode: Preset 4 - Input: Bosphorus 4K Frames Per Second > Higher Is Better a . 9.892 |==================================================================== b . 9.793 |=================================================================== c . 9.889 |==================================================================== d . 9.866 |==================================================================== SVT-AV1 2.0 Encoder Mode: Preset 8 - Input: Bosphorus 4K Frames Per Second > Higher Is Better a . 94.51 |==================================================================== b . 93.73 |=================================================================== c . 93.93 |==================================================================== d . 94.56 |==================================================================== SVT-AV1 2.0 Encoder Mode: Preset 12 - Input: Bosphorus 4K Frames Per Second > Higher Is Better a . 188.56 |================================================================ b . 195.91 |=================================================================== c . 194.07 |================================================================== d . 195.86 |=================================================================== SVT-AV1 2.0 Encoder Mode: Preset 13 - Input: Bosphorus 4K Frames Per Second > Higher Is Better a . 192.57 |=================================================================== b . 193.29 |=================================================================== c . 192.64 |=================================================================== d . 187.63 |================================================================= SVT-AV1 2.0 Encoder Mode: Preset 4 - Input: Bosphorus 1080p Frames Per Second > Higher Is Better a . 27.48 |==================================================================== b . 27.28 |=================================================================== c . 27.52 |==================================================================== d . 27.24 |=================================================================== SVT-AV1 2.0 Encoder Mode: Preset 8 - Input: Bosphorus 1080p Frames Per Second > Higher Is Better a . 193.76 |================================================================== b . 194.77 |=================================================================== c . 189.61 |================================================================= d . 195.61 |=================================================================== SVT-AV1 2.0 Encoder Mode: Preset 12 - Input: Bosphorus 1080p Frames Per Second > Higher Is Better a . 616.58 |================================================================== b . 615.54 |================================================================== c . 625.82 |=================================================================== d . 621.44 |=================================================================== SVT-AV1 2.0 Encoder Mode: Preset 13 - Input: Bosphorus 1080p Frames Per Second > Higher Is Better a . 662.49 |================================================================== b . 653.69 |================================================================= c . 671.83 |=================================================================== d . 670.14 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 21.85 |==================================================================== b . 21.82 |==================================================================== c . 21.86 |==================================================================== d . 21.80 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 365.34 |=================================================================== b . 365.98 |=================================================================== c . 365.58 |=================================================================== d . 366.68 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 17.23 |==================================================================== b . 17.23 |==================================================================== c . 17.23 |==================================================================== d . 17.21 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 58.02 |==================================================================== b . 58.02 |==================================================================== c . 58.02 |==================================================================== d . 58.09 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 887.44 |=================================================================== b . 886.18 |=================================================================== c . 884.66 |=================================================================== d . 886.78 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 8.9970 |=================================================================== b . 9.0094 |=================================================================== c . 9.0252 |=================================================================== d . 9.0040 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 252.36 |=================================================================== b . 251.61 |=================================================================== c . 251.25 |=================================================================== d . 253.03 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 3.9594 |=================================================================== b . 3.9707 |=================================================================== c . 3.9766 |=================================================================== d . 3.9490 |=================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 274.26 |=================================================================== b . 274.47 |=================================================================== c . 274.47 |=================================================================== d . 274.46 |=================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 29.15 |==================================================================== b . 29.12 |==================================================================== c . 29.12 |==================================================================== d . 29.12 |==================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 170.31 |=================================================================== b . 171.49 |=================================================================== c . 171.48 |=================================================================== d . 171.36 |=================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 5.8648 |=================================================================== b . 5.8243 |=================================================================== c . 5.8245 |=================================================================== d . 5.8285 |=================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 2253.86 |================================================================== b . 2269.51 |================================================================== c . 2254.22 |================================================================== d . 2236.92 |================================================================= Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 3.5332 |================================================================== b . 3.5087 |================================================================== c . 3.5324 |================================================================== d . 3.5604 |=================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 1134.77 |================================================================== b . 1131.64 |================================================================== c . 1133.50 |================================================================== d . 1132.32 |================================================================== Neural Magic DeepSparse 1.7 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 0.8788 |=================================================================== b . 0.8810 |=================================================================== c . 0.8796 |=================================================================== d . 0.8806 |=================================================================== Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 3.4391 |=================================================================== b . 3.4406 |=================================================================== c . 3.4298 |=================================================================== d . 3.4284 |=================================================================== Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 2252.46 |================================================================== b . 2251.61 |================================================================== c . 2258.14 |================================================================== d . 2259.26 |================================================================== Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 11.12 |==================================================================== b . 11.02 |=================================================================== c . 11.08 |==================================================================== d . 11.01 |=================================================================== Neural Magic DeepSparse 1.7 Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 89.87 |=================================================================== b . 90.75 |==================================================================== c . 90.23 |==================================================================== d . 90.77 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 274.59 |=================================================================== b . 274.39 |=================================================================== c . 274.28 |=================================================================== d . 274.60 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 29.11 |==================================================================== b . 29.13 |==================================================================== c . 29.14 |==================================================================== d . 29.11 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 170.96 |=================================================================== b . 171.31 |=================================================================== c . 171.28 |=================================================================== d . 171.36 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 5.8423 |=================================================================== b . 5.8302 |=================================================================== c . 5.8312 |=================================================================== d . 5.8281 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 129.44 |=================================================================== b . 128.96 |=================================================================== c . 129.10 |=================================================================== d . 129.46 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 61.77 |==================================================================== b . 62.00 |==================================================================== c . 61.94 |==================================================================== d . 61.76 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 87.36 |==================================================================== b . 87.55 |==================================================================== c . 87.63 |==================================================================== d . 87.62 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 11.44 |==================================================================== b . 11.42 |==================================================================== c . 11.41 |==================================================================== d . 11.41 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 188.33 |=================================================================== b . 188.15 |=================================================================== c . 188.21 |=================================================================== d . 188.44 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 42.45 |==================================================================== b . 42.49 |==================================================================== c . 42.48 |==================================================================== d . 42.42 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 113.47 |================================================================== b . 114.41 |=================================================================== c . 113.88 |================================================================== d . 114.83 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 8.8070 |=================================================================== b . 8.7347 |================================================================== c . 8.7752 |=================================================================== d . 8.7026 |================================================================== Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 39.57 |==================================================================== b . 39.47 |==================================================================== c . 39.47 |==================================================================== d . 39.45 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 202.02 |=================================================================== b . 202.51 |=================================================================== c . 202.37 |=================================================================== d . 202.46 |=================================================================== Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 29.77 |==================================================================== b . 29.82 |==================================================================== c . 29.79 |==================================================================== d . 29.79 |==================================================================== Neural Magic DeepSparse 1.7 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 33.57 |==================================================================== b . 33.52 |==================================================================== c . 33.55 |==================================================================== d . 33.55 |==================================================================== Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 396.58 |=================================================================== b . 396.44 |=================================================================== c . 397.59 |=================================================================== d . 397.57 |=================================================================== Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 20.15 |==================================================================== b . 20.16 |==================================================================== c . 20.10 |==================================================================== d . 20.10 |==================================================================== Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 89.71 |=================================================================== b . 90.50 |==================================================================== c . 89.81 |=================================================================== d . 90.39 |==================================================================== Neural Magic DeepSparse 1.7 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 11.14 |==================================================================== b . 11.04 |=================================================================== c . 11.12 |==================================================================== d . 11.05 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 21.92 |==================================================================== b . 21.90 |==================================================================== c . 21.90 |==================================================================== d . 21.95 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 363.53 |=================================================================== b . 364.43 |=================================================================== c . 363.76 |=================================================================== d . 363.59 |=================================================================== Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 17.22 |==================================================================== b . 17.20 |==================================================================== c . 17.18 |==================================================================== d . 17.20 |==================================================================== Neural Magic DeepSparse 1.7 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 58.07 |==================================================================== b . 58.14 |==================================================================== c . 58.19 |==================================================================== d . 58.14 |====================================================================