2024 year AMD Ryzen Threadripper PRO 5965WX 24-Cores testing with a ASUS Pro WS WRX80E-SAGE SE WIFI (1201 BIOS) and ASUS NVIDIA NV106 2GB on Ubuntu 23.10 via the Phoronix Test Suite. a: Processor: AMD Ryzen Threadripper PRO 5965WX 24-Cores @ 3.80GHz (24 Cores / 48 Threads), Motherboard: ASUS Pro WS WRX80E-SAGE SE WIFI (1201 BIOS), Chipset: AMD Starship/Matisse, Memory: 8 x 16GB DDR4-2133MT/s Corsair CMK32GX4M2E3200C16, Disk: 2048GB SOLIDIGM SSDPFKKW020X7, Graphics: ASUS NVIDIA NV106 2GB, Audio: AMD Starship/Matisse, Monitor: VA2431, Network: 2 x Intel X550 + Intel Wi-Fi 6 AX200 OS: Ubuntu 23.10, Kernel: 6.5.0-13-generic (x86_64), Desktop: GNOME Shell 45.0, Display Server: X Server + Wayland, Display Driver: nouveau, OpenGL: 4.3 Mesa 23.2.1-1ubuntu3, Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080 b: Processor: AMD Ryzen Threadripper PRO 5965WX 24-Cores @ 3.80GHz (24 Cores / 48 Threads), Motherboard: ASUS Pro WS WRX80E-SAGE SE WIFI (1201 BIOS), Chipset: AMD Starship/Matisse, Memory: 8 x 16GB DDR4-2133MT/s Corsair CMK32GX4M2E3200C16, Disk: 2048GB SOLIDIGM SSDPFKKW020X7, Graphics: ASUS NVIDIA NV106 2GB, Audio: AMD Starship/Matisse, Monitor: VA2431, Network: 2 x Intel X550 + Intel Wi-Fi 6 AX200 OS: Ubuntu 23.10, Kernel: 6.5.0-13-generic (x86_64), Desktop: GNOME Shell 45.0, Display Server: X Server + Wayland, Display Driver: nouveau, OpenGL: 4.3 Mesa 23.2.1-1ubuntu3, Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080 c: Processor: AMD Ryzen Threadripper PRO 5965WX 24-Cores @ 3.80GHz (24 Cores / 48 Threads), Motherboard: ASUS Pro WS WRX80E-SAGE SE WIFI (1201 BIOS), Chipset: AMD Starship/Matisse, Memory: 8 x 16GB DDR4-2133MT/s Corsair CMK32GX4M2E3200C16, Disk: 2048GB SOLIDIGM SSDPFKKW020X7, Graphics: ASUS NVIDIA NV106 2GB, Audio: AMD Starship/Matisse, Monitor: VA2431, Network: 2 x Intel X550 + Intel Wi-Fi 6 AX200 OS: Ubuntu 23.10, Kernel: 6.5.0-13-generic (x86_64), Desktop: GNOME Shell 45.0, Display Server: X Server + Wayland, Display Driver: nouveau, OpenGL: 4.3 Mesa 23.2.1-1ubuntu3, Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080 d: Processor: AMD Ryzen Threadripper PRO 5965WX 24-Cores @ 3.80GHz (24 Cores / 48 Threads), Motherboard: ASUS Pro WS WRX80E-SAGE SE WIFI (1201 BIOS), Chipset: AMD Starship/Matisse, Memory: 8 x 16GB DDR4-2133MT/s Corsair CMK32GX4M2E3200C16, Disk: 2048GB SOLIDIGM SSDPFKKW020X7, Graphics: ASUS NVIDIA NV106 2GB, Audio: AMD Starship/Matisse, Monitor: VA2431, Network: 2 x Intel X550 + Intel Wi-Fi 6 AX200 OS: Ubuntu 23.10, Kernel: 6.5.0-13-generic (x86_64), Desktop: GNOME Shell 45.0, Display Server: X Server + Wayland, Display Driver: nouveau, OpenGL: 4.3 Mesa 23.2.1-1ubuntu3, Compiler: GCC 13.2.0, File-System: ext4, Screen Resolution: 1920x1080 LeelaChessZero 0.30 Backend: BLAS Nodes Per Second > Higher Is Better a . 173 |====================================================== b . 219 |==================================================================== c . 225 |====================================================================== d . 213 |================================================================== LeelaChessZero 0.30 Backend: Eigen Nodes Per Second > Higher Is Better a . 121 |======================================================= b . 146 |================================================================== c . 154 |====================================================================== d . 151 |===================================================================== Quicksilver 20230818 Input: CTS2 Figure Of Merit > Higher Is Better a . 20680000 |================================================================= b . 20646667 |================================================================= c . 20620000 |================================================================= d . 20600000 |================================================================= Quicksilver 20230818 Input: CORAL2 P1 Figure Of Merit > Higher Is Better a . 24210000 |================================================================= b . 24230000 |================================================================= c . 24240000 |================================================================= d . 24290000 |================================================================= Quicksilver 20230818 Input: CORAL2 P2 Figure Of Merit > Higher Is Better a . 24030000 |================================================================= b . 24026667 |================================================================= c . 23890000 |================================================================= d . 23840000 |================================================================ CacheBench Test: Read MB/s > Higher Is Better a . 11543.37 |================================================================= b . 11543.16 |================================================================= c . 11543.10 |================================================================= d . 11543.49 |================================================================= CacheBench Test: Write MB/s > Higher Is Better a . 69134.68 |================================================================= b . 69140.53 |================================================================= c . 69142.44 |================================================================= d . 69142.19 |================================================================= CacheBench Test: Read / Modify / Write MB/s > Higher Is Better a . 130857.58 |================================================================ b . 130069.85 |================================================================ c . 130806.25 |================================================================ d . 130851.31 |================================================================ LZ4 Compression 1.9.4 Compression Level: 1 - Compression Speed MB/s > Higher Is Better a . 828.78 |=================================================================== b . 829.36 |=================================================================== c . 829.15 |=================================================================== d . 830.37 |=================================================================== LZ4 Compression 1.9.4 Compression Level: 1 - Decompression Speed MB/s > Higher Is Better a . 5019.5 |=================================================================== b . 5020.0 |=================================================================== c . 5023.2 |=================================================================== d . 5019.5 |=================================================================== LZ4 Compression 1.9.4 Compression Level: 3 - Compression Speed MB/s > Higher Is Better a . 131.24 |=================================================================== b . 131.10 |=================================================================== c . 131.40 |=================================================================== d . 131.61 |=================================================================== LZ4 Compression 1.9.4 Compression Level: 3 - Decompression Speed MB/s > Higher Is Better a . 4595.9 |=================================================================== b . 4597.9 |=================================================================== c . 4598.0 |=================================================================== d . 4596.7 |=================================================================== LZ4 Compression 1.9.4 Compression Level: 9 - Compression Speed MB/s > Higher Is Better a . 44.28 |================================================================== b . 44.48 |================================================================== c . 45.49 |==================================================================== d . 44.52 |=================================================================== LZ4 Compression 1.9.4 Compression Level: 9 - Decompression Speed MB/s > Higher Is Better a . 4840.5 |=================================================================== b . 4841.4 |=================================================================== c . 4842.4 |=================================================================== d . 4844.5 |=================================================================== rav1e 0.7 Speed: 1 Frames Per Second > Higher Is Better a . 1.044 |=================================================================== b . 1.048 |==================================================================== c . 1.044 |=================================================================== d . 1.054 |==================================================================== rav1e 0.7 Speed: 5 Frames Per Second > Higher Is Better a . 3.747 |================================================================= b . 3.791 |================================================================== c . 3.769 |================================================================== d . 3.891 |==================================================================== rav1e 0.7 Speed: 6 Frames Per Second > Higher Is Better a . 5.261 |==================================================================== b . 5.292 |==================================================================== c . 5.282 |==================================================================== d . 5.191 |=================================================================== rav1e 0.7 Speed: 10 Frames Per Second > Higher Is Better a . 10.63 |================================================================== b . 10.89 |=================================================================== c . 10.96 |==================================================================== d . 11.02 |==================================================================== SVT-AV1 1.8 Encoder Mode: Preset 4 - Input: Bosphorus 4K Frames Per Second > Higher Is Better a . 6.677 |==================================================================== b . 6.669 |==================================================================== c . 6.678 |==================================================================== d . 6.633 |==================================================================== SVT-AV1 1.8 Encoder Mode: Preset 8 - Input: Bosphorus 4K Frames Per Second > Higher Is Better a . 61.53 |==================================================================== b . 61.83 |==================================================================== c . 61.47 |==================================================================== d . 60.95 |=================================================================== SVT-AV1 1.8 Encoder Mode: Preset 12 - Input: Bosphorus 4K Frames Per Second > Higher Is Better a . 190.92 |================================================================== b . 190.40 |================================================================== c . 192.16 |=================================================================== d . 192.68 |=================================================================== SVT-AV1 1.8 Encoder Mode: Preset 13 - Input: Bosphorus 4K Frames Per Second > Higher Is Better a . 190.79 |================================================================== b . 192.73 |=================================================================== c . 189.25 |================================================================== d . 192.60 |=================================================================== SVT-AV1 1.8 Encoder Mode: Preset 4 - Input: Bosphorus 1080p Frames Per Second > Higher Is Better a . 18.80 |==================================================================== b . 18.68 |=================================================================== c . 18.41 |================================================================== d . 18.92 |==================================================================== SVT-AV1 1.8 Encoder Mode: Preset 8 - Input: Bosphorus 1080p Frames Per Second > Higher Is Better a . 122.95 |=================================================================== b . 123.29 |=================================================================== c . 122.95 |=================================================================== d . 122.62 |=================================================================== SVT-AV1 1.8 Encoder Mode: Preset 12 - Input: Bosphorus 1080p Frames Per Second > Higher Is Better a . 501.42 |================================================================== b . 506.60 |=================================================================== c . 501.16 |================================================================== d . 506.17 |=================================================================== SVT-AV1 1.8 Encoder Mode: Preset 13 - Input: Bosphorus 1080p Frames Per Second > Higher Is Better a . 543.55 |=============================================================== b . 573.04 |================================================================== c . 580.47 |=================================================================== d . 565.91 |================================================================= Y-Cruncher 0.8.3 Pi Digits To Calculate: 1B Seconds < Lower Is Better a . 15.55 |==================================================================== b . 15.50 |==================================================================== c . 15.53 |==================================================================== d . 15.50 |==================================================================== Y-Cruncher 0.8.3 Pi Digits To Calculate: 500M Seconds < Lower Is Better a . 7.325 |==================================================================== b . 7.349 |==================================================================== c . 7.301 |==================================================================== d . 7.297 |==================================================================== PyTorch 2.1 Device: CPU - Batch Size: 1 - Model: ResNet-50 batches/sec > Higher Is Better a . 40.68 |==================================================================== b . 40.42 |=================================================================== c . 40.82 |==================================================================== d . 40.35 |=================================================================== PyTorch 2.1 Device: CPU - Batch Size: 16 - Model: ResNet-50 batches/sec > Higher Is Better a . 32.50 |==================================================================== b . 32.10 |=================================================================== c . 31.65 |================================================================== d . 32.21 |=================================================================== PyTorch 2.1 Device: CPU - Batch Size: 256 - Model: ResNet-50 batches/sec > Higher Is Better a . 31.92 |==================================================================== b . 32.15 |==================================================================== c . 31.59 |=================================================================== d . 32.07 |==================================================================== TensorFlow 2.12 Device: CPU - Batch Size: 1 - Model: VGG-16 images/sec > Higher Is Better a . 2.72 |===================================================================== b . 2.70 |==================================================================== c . 2.72 |===================================================================== d . 2.69 |==================================================================== TensorFlow 2.12 Device: CPU - Batch Size: 1 - Model: AlexNet images/sec > Higher Is Better a . 6.26 |===================================================================== b . 6.23 |===================================================================== c . 6.23 |===================================================================== d . 6.21 |==================================================================== TensorFlow 2.12 Device: CPU - Batch Size: 16 - Model: VGG-16 images/sec > Higher Is Better a . 8.51 |===================================================================== b . 8.46 |===================================================================== c . 8.48 |===================================================================== d . 8.51 |===================================================================== TensorFlow 2.12 Device: CPU - Batch Size: 16 - Model: AlexNet images/sec > Higher Is Better a . 100.44 |=================================================================== b . 100.01 |=================================================================== c . 100.08 |=================================================================== d . 99.90 |=================================================================== TensorFlow 2.12 Device: CPU - Batch Size: 1 - Model: GoogLeNet images/sec > Higher Is Better a . 9.94 |========================================= b . 9.74 |======================================== c . 16.49 |==================================================================== d . 9.73 |======================================== TensorFlow 2.12 Device: CPU - Batch Size: 1 - Model: ResNet-50 images/sec > Higher Is Better a . 8.85 |===================================================================== b . 8.79 |==================================================================== c . 8.85 |===================================================================== d . 8.89 |===================================================================== TensorFlow 2.12 Device: CPU - Batch Size: 16 - Model: GoogLeNet images/sec > Higher Is Better a . 60.85 |==================================================================== b . 60.18 |=================================================================== c . 60.04 |=================================================================== d . 59.82 |=================================================================== TensorFlow 2.12 Device: CPU - Batch Size: 16 - Model: ResNet-50 images/sec > Higher Is Better a . 19.87 |==================================================================== b . 19.45 |=================================================================== c . 19.61 |=================================================================== d . 19.81 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 26.84 |==================================================================== b . 26.65 |==================================================================== c . 26.67 |==================================================================== d . 26.69 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 446.63 |=================================================================== b . 448.91 |=================================================================== c . 449.30 |=================================================================== d . 449.16 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 18.39 |==================================================================== b . 18.33 |==================================================================== c . 18.36 |==================================================================== d . 18.40 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 54.37 |==================================================================== b . 54.54 |==================================================================== c . 54.46 |==================================================================== d . 54.35 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 685.12 |=================================================================== b . 681.00 |=================================================================== c . 683.25 |=================================================================== d . 681.53 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 17.50 |==================================================================== b . 17.60 |==================================================================== c . 17.54 |==================================================================== d . 17.59 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 189.91 |================================================================== b . 193.11 |=================================================================== c . 192.98 |=================================================================== d . 189.27 |================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 5.2624 |=================================================================== b . 5.1753 |================================================================== c . 5.1787 |================================================================== d . 5.2804 |=================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 307.01 |=================================================================== b . 304.77 |=================================================================== c . 306.50 |=================================================================== d . 305.86 |=================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 39.04 |=================================================================== b . 39.33 |==================================================================== c . 39.12 |==================================================================== d . 39.19 |==================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 157.23 |=================================================================== b . 155.48 |================================================================== c . 155.40 |================================================================== d . 155.20 |================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 6.3518 |================================================================== b . 6.4230 |=================================================================== c . 6.4266 |=================================================================== d . 6.4345 |=================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 2012.21 |================================================================== b . 1962.70 |================================================================ c . 1970.61 |================================================================= d . 1962.16 |================================================================ Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 5.9507 |================================================================= b . 6.1010 |=================================================================== c . 6.0747 |=================================================================== d . 6.1024 |=================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 757.16 |================================================================== b . 752.78 |================================================================== c . 765.99 |=================================================================== d . 757.27 |================================================================== Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 1.3175 |=================================================================== b . 1.3252 |=================================================================== c . 1.3025 |================================================================== d . 1.3175 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 150.04 |=================================================================== b . 148.24 |================================================================== c . 148.76 |================================================================== d . 149.27 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 79.89 |=================================================================== b . 80.85 |==================================================================== c . 80.57 |==================================================================== d . 80.27 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 98.52 |==================================================================== b . 98.56 |==================================================================== c . 98.65 |==================================================================== d . 98.77 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 10.14 |==================================================================== b . 10.14 |==================================================================== c . 10.13 |==================================================================== d . 10.12 |==================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 32.52 |==================================================================== b . 32.36 |==================================================================== c . 32.42 |==================================================================== d . 32.38 |==================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 368.31 |=================================================================== b . 369.77 |=================================================================== c . 369.62 |=================================================================== d . 369.53 |=================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 17.16 |==================================================================== b . 17.13 |==================================================================== c . 17.24 |==================================================================== d . 17.12 |==================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 58.25 |==================================================================== b . 58.35 |==================================================================== c . 58.00 |==================================================================== d . 58.41 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 306.99 |=================================================================== b . 305.38 |=================================================================== c . 306.52 |=================================================================== d . 299.83 |================================================================= Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 39.05 |================================================================== b . 39.26 |=================================================================== c . 39.12 |=================================================================== d . 39.98 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 156.97 |=================================================================== b . 155.54 |================================================================== c . 155.62 |================================================================== d . 155.19 |================================================================== Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 6.3619 |================================================================== b . 6.4208 |=================================================================== c . 6.4178 |=================================================================== d . 6.4346 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 151.47 |=================================================================== b . 149.78 |================================================================== c . 149.95 |================================================================== d . 149.76 |================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 79.16 |=================================================================== b . 80.05 |==================================================================== c . 79.89 |==================================================================== d . 80.05 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 98.91 |==================================================================== b . 98.75 |==================================================================== c . 98.50 |==================================================================== d . 98.89 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 10.10 |==================================================================== b . 10.12 |==================================================================== c . 10.15 |==================================================================== d . 10.11 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 224.18 |=================================================================== b . 221.71 |================================================================== c . 223.05 |=================================================================== d . 222.72 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 53.47 |=================================================================== b . 54.06 |==================================================================== c . 53.74 |==================================================================== d . 53.82 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 112.09 |=================================================================== b . 111.29 |=================================================================== c . 111.59 |=================================================================== d . 111.50 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 8.9109 |=================================================================== b . 8.9755 |=================================================================== c . 8.9520 |=================================================================== d . 8.9586 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 30.49 |==================================================================== b . 30.28 |==================================================================== c . 30.45 |==================================================================== d . 30.32 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 392.96 |=================================================================== b . 394.78 |=================================================================== c . 393.83 |=================================================================== d . 394.38 |=================================================================== Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 21.73 |==================================================================== b . 21.47 |=================================================================== c . 21.69 |==================================================================== d . 21.67 |==================================================================== Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 46.00 |=================================================================== b . 46.55 |==================================================================== c . 46.09 |=================================================================== d . 46.13 |=================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 335.35 |=================================================================== b . 333.37 |=================================================================== c . 333.64 |=================================================================== d . 334.64 |=================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 35.76 |==================================================================== b . 35.97 |==================================================================== c . 35.93 |==================================================================== d . 35.84 |==================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 76.14 |==================================================================== b . 75.51 |=================================================================== c . 75.99 |==================================================================== d . 75.71 |==================================================================== Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 13.13 |=================================================================== b . 13.24 |==================================================================== c . 13.15 |==================================================================== d . 13.20 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream items/sec > Higher Is Better a . 26.91 |==================================================================== b . 26.67 |=================================================================== c . 26.71 |=================================================================== d . 26.54 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream ms/batch < Lower Is Better a . 445.26 |================================================================== b . 448.04 |=================================================================== c . 448.90 |=================================================================== d . 448.03 |=================================================================== Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream items/sec > Higher Is Better a . 18.47 |==================================================================== b . 18.40 |==================================================================== c . 18.46 |==================================================================== d . 18.44 |==================================================================== Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream ms/batch < Lower Is Better a . 54.12 |==================================================================== b . 54.33 |==================================================================== c . 54.15 |==================================================================== d . 54.20 |==================================================================== Speedb 2.7 Test: Random Fill Op/s > Higher Is Better a . 558330 |=================================================================== b . 554997 |=================================================================== c . 557348 |=================================================================== d . 556675 |=================================================================== Speedb 2.7 Test: Random Read Op/s > Higher Is Better a . 148134848 |================================================================ b . 147432214 |================================================================ c . 146285738 |=============================================================== d . 146473036 |=============================================================== Speedb 2.7 Test: Update Random Op/s > Higher Is Better a . 431692 |=================================================================== b . 418848 |================================================================= c . 423788 |================================================================== d . 417457 |================================================================= Speedb 2.7 Test: Sequential Fill Op/s > Higher Is Better a . 620607 |=================================================================== b . 618776 |=================================================================== c . 604758 |================================================================= d . 612428 |================================================================== Speedb 2.7 Test: Random Fill Sync Op/s > Higher Is Better a . 47488 |==================================================================== b . 47708 |==================================================================== c . 47373 |==================================================================== d . 47267 |=================================================================== Speedb 2.7 Test: Read While Writing Op/s > Higher Is Better a . 7004007 |================================================================= b . 7070407 |================================================================== c . 7047502 |================================================================== d . 6896007 |================================================================ Speedb 2.7 Test: Read Random Write Random Op/s > Higher Is Better a . 2327911 |================================================================== b . 2307686 |================================================================= c . 2320670 |================================================================== d . 2316804 |================================================================== Llama.cpp b1808 Model: llama-2-7b.Q4_0.gguf Tokens Per Second > Higher Is Better a . 20.76 |=================================================================== b . 20.95 |==================================================================== c . 20.74 |=================================================================== d . 20.68 |=================================================================== Llama.cpp b1808 Model: llama-2-13b.Q4_0.gguf Tokens Per Second > Higher Is Better a . 11.64 |==================================================================== b . 11.32 |================================================================== c . 11.27 |================================================================== d . 11.25 |================================================================== Llama.cpp b1808 Model: llama-2-70b-chat.Q5_0.gguf Tokens Per Second > Higher Is Better a . 1.94 |===================================================================== b . 1.94 |===================================================================== c . 1.95 |===================================================================== d . 1.95 |===================================================================== Llamafile 0.6 Test: llava-v1.5-7b-q4 - Acceleration: CPU Tokens Per Second > Higher Is Better a . 17.22 |==================================================================== b . 17.26 |==================================================================== c . 17.30 |==================================================================== d . 17.25 |==================================================================== Llamafile 0.6 Test: mistral-7b-instruct-v0.2.Q8_0 - Acceleration: CPU Tokens Per Second > Higher Is Better a . 10.13 |==================================================================== b . 10.15 |==================================================================== c . 10.14 |==================================================================== d . 10.13 |==================================================================== Llamafile 0.6 Test: wizardcoder-python-34b-v1.0.Q6_K - Acceleration: CPU Tokens Per Second > Higher Is Better a . 3.25 |===================================================================== b . 3.25 |===================================================================== c . 3.25 |===================================================================== d . 3.25 |=====================================================================