2024 year AMD Ryzen Threadripper PRO 5965WX 24-Cores testing with a ASUS Pro WS WRX80E-SAGE SE WIFI (1201 BIOS) and ASUS NVIDIA NV106 2GB on Ubuntu 23.10 via the Phoronix Test Suite.
HTML result view exported from: https://openbenchmarking.org/result/2402040-NE-2024YEAR116 .
2024 year Processor Motherboard Chipset Memory Disk Graphics Audio Monitor Network OS Kernel Desktop Display Server Display Driver OpenGL Compiler File-System Screen Resolution a b c d AMD Ryzen Threadripper PRO 5965WX 24-Cores @ 3.80GHz (24 Cores / 48 Threads) ASUS Pro WS WRX80E-SAGE SE WIFI (1201 BIOS) AMD Starship/Matisse 8 x 16GB DDR4-2133MT/s Corsair CMK32GX4M2E3200C16 2048GB SOLIDIGM SSDPFKKW020X7 ASUS NVIDIA NV106 2GB AMD Starship/Matisse VA2431 2 x Intel X550 + Intel Wi-Fi 6 AX200 Ubuntu 23.10 6.5.0-13-generic (x86_64) GNOME Shell 45.0 X Server + Wayland nouveau 4.3 Mesa 23.2.1-1ubuntu3 GCC 13.2.0 ext4 1920x1080 OpenBenchmarking.org Kernel Details - Transparent Huge Pages: madvise Compiler Details - --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-cet --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-serialization=2 --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-defaulted --enable-offload-targets=nvptx-none=/build/gcc-13-XYspKM/gcc-13-13.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-13-XYspKM/gcc-13-13.2.0/debian/tmp-gcn/usr --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details - Scaling Governor: acpi-cpufreq schedutil (Boost: Enabled) - CPU Microcode: 0xa008205 Python Details - Python 3.11.6 Security Details - gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_rstack_overflow: Mitigation of safe RET no microcode + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected
2024 year lczero: BLAS lczero: Eigen quicksilver: CTS2 quicksilver: CORAL2 P1 quicksilver: CORAL2 P2 cachebench: Read cachebench: Write cachebench: Read / Modify / Write compress-lz4: 1 - Compression Speed compress-lz4: 1 - Decompression Speed compress-lz4: 3 - Compression Speed compress-lz4: 3 - Decompression Speed compress-lz4: 9 - Compression Speed compress-lz4: 9 - Decompression Speed rav1e: 1 rav1e: 5 rav1e: 6 rav1e: 10 svt-av1: Preset 4 - Bosphorus 4K svt-av1: Preset 8 - Bosphorus 4K svt-av1: Preset 12 - Bosphorus 4K svt-av1: Preset 13 - Bosphorus 4K svt-av1: Preset 4 - Bosphorus 1080p svt-av1: Preset 8 - Bosphorus 1080p svt-av1: Preset 12 - Bosphorus 1080p svt-av1: Preset 13 - Bosphorus 1080p y-cruncher: 1B y-cruncher: 500M pytorch: CPU - 1 - ResNet-50 pytorch: CPU - 16 - ResNet-50 pytorch: CPU - 256 - ResNet-50 tensorflow: CPU - 1 - VGG-16 tensorflow: CPU - 1 - AlexNet tensorflow: CPU - 16 - VGG-16 tensorflow: CPU - 16 - AlexNet tensorflow: CPU - 1 - GoogLeNet tensorflow: CPU - 1 - ResNet-50 tensorflow: CPU - 16 - GoogLeNet tensorflow: CPU - 16 - ResNet-50 deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Asynchronous Multi-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: NLP Document Classification, oBERT base uncased on IMDB - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Asynchronous Multi-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Synchronous Single-Stream deepsparse: ResNet-50, Baseline - Asynchronous Multi-Stream deepsparse: ResNet-50, Baseline - Asynchronous Multi-Stream deepsparse: ResNet-50, Baseline - Synchronous Single-Stream deepsparse: ResNet-50, Baseline - Synchronous Single-Stream deepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Stream deepsparse: ResNet-50, Sparse INT8 - Asynchronous Multi-Stream deepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Stream deepsparse: ResNet-50, Sparse INT8 - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Asynchronous Multi-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream deepsparse: CV Classification, ResNet-50 ImageNet - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Asynchronous Multi-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Stream deepsparse: CV Detection, YOLOv5s COCO, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Asynchronous Multi-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream deepsparse: NLP Text Classification, DistilBERT mnli - Synchronous Single-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Asynchronous Multi-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Stream deepsparse: CV Segmentation, 90% Pruned YOLACT Pruned - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Asynchronous Multi-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Stream deepsparse: BERT-Large, NLP Question Answering, Sparse INT8 - Synchronous Single-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Asynchronous Multi-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream deepsparse: NLP Token Classification, BERT base uncased conll2003 - Synchronous Single-Stream speedb: Rand Fill speedb: Rand Read speedb: Update Rand speedb: Seq Fill speedb: Rand Fill Sync speedb: Read While Writing speedb: Read Rand Write Rand llama-cpp: llama-2-7b.Q4_0.gguf llama-cpp: llama-2-13b.Q4_0.gguf llama-cpp: llama-2-70b-chat.Q5_0.gguf llamafile: llava-v1.5-7b-q4 - CPU llamafile: mistral-7b-instruct-v0.2.Q8_0 - CPU llamafile: wizardcoder-python-34b-v1.0.Q6_K - CPU a b c d 173 121 20680000 24210000 24030000 11543.372321 69134.680498 130857.577562 828.78 5019.5 131.24 4595.9 44.28 4840.5 1.044 3.747 5.261 10.634 6.677 61.53 190.916 190.794 18.803 122.948 501.42 543.554 15.545 7.325 40.68 32.50 31.92 2.72 6.26 8.51 100.44 9.94 8.85 60.85 19.87 26.8394 446.6313 18.3872 54.3716 685.1217 17.495 189.9093 5.2624 307.0146 39.0436 157.2286 6.3518 2012.2062 5.9507 757.1631 1.3175 150.0421 79.8855 98.516 10.1424 32.5188 368.3051 17.1616 58.254 306.9919 39.0493 156.9701 6.3619 151.4666 79.1618 98.9102 10.1046 224.1802 53.4699 112.0914 8.9109 30.489 392.9588 21.7296 45.9976 335.3479 35.762 76.1394 13.129 26.9119 445.2565 18.4727 54.1194 558330 148134848 431692 620607 47488 7004007 2327911 20.76 11.64 1.94 17.22 10.13 3.25 219 146 20646667 24230000 24026667 11543.164687 69140.530992 130069.848338 829.36 5020.0 131.10 4597.9 44.48 4841.4 1.048 3.791 5.292 10.885 6.669 61.830 190.402 192.726 18.682 123.293 506.596 573.042 15.497 7.349 40.42 32.10 32.15 2.70 6.23 8.46 100.01 9.74 8.79 60.18 19.45 26.6478 448.9079 18.3317 54.5365 681.0038 17.5999 193.1106 5.1753 304.7712 39.3339 155.4837 6.4230 1962.7047 6.1010 752.7784 1.3252 148.2388 80.8465 98.5635 10.1362 32.3621 369.7720 17.1337 58.3490 305.3795 39.2555 155.5402 6.4208 149.7818 80.0470 98.7456 10.1213 221.7053 54.0612 111.2892 8.9755 30.2809 394.7759 21.4743 46.5453 333.3674 35.9726 75.5067 13.2388 26.6672 448.0408 18.4000 54.3340 554997 147432214 418848 618776 47708 7070407 2307686 20.95 11.32 1.94 17.26 10.15 3.25 225 154 20620000 24240000 23890000 11543.096362 69142.435503 130806.245683 829.15 5023.2 131.4 4598 45.49 4842.4 1.044 3.769 5.282 10.957 6.678 61.47 192.157 189.252 18.409 122.95 501.156 580.467 15.532 7.301 40.82 31.65 31.59 2.72 6.23 8.48 100.08 16.49 8.85 60.04 19.61 26.6741 449.2956 18.3589 54.4554 683.2461 17.5377 192.9801 5.1787 306.5028 39.1233 155.403 6.4266 1970.6106 6.0747 765.9923 1.3025 148.7561 80.5729 98.6526 10.1277 32.4212 369.6228 17.2376 57.9978 306.5175 39.1181 155.6182 6.4178 149.9507 79.8946 98.4979 10.146 223.0515 53.7417 111.5864 8.952 30.451 393.8346 21.6853 46.0912 333.6358 35.9278 75.9898 13.1545 26.705 448.9031 18.4635 54.1474 557348 146285738 423788 604758 47373 7047502 2320670 20.74 11.27 1.95 17.3 10.14 3.25 213 151 20600000 24290000 23840000 11543.486939 69142.188854 130851.30507 830.37 5019.5 131.61 4596.7 44.52 4844.5 1.054 3.891 5.191 11.022 6.633 60.95 192.683 192.603 18.922 122.616 506.174 565.906 15.501 7.297 40.35 32.21 32.07 2.69 6.21 8.51 99.9 9.73 8.89 59.82 19.81 26.6902 449.1605 18.3959 54.3471 681.5303 17.5865 189.2673 5.2804 305.8614 39.1902 155.2016 6.4345 1962.1551 6.1024 757.2715 1.3175 149.2683 80.2684 98.7671 10.1151 32.3762 369.53 17.1162 58.4094 299.8295 39.977 155.1926 6.4346 149.7647 80.052 98.8912 10.1068 222.7232 53.8237 111.4965 8.9586 30.3159 394.3756 21.6655 46.1334 334.6424 35.837 75.7073 13.204 26.5435 448.0349 18.4443 54.204 556675 146473036 417457 612428 47267 6896007 2316804 20.68 11.25 1.95 17.25 10.13 3.25 OpenBenchmarking.org
LeelaChessZero Backend: BLAS OpenBenchmarking.org Nodes Per Second, More Is Better LeelaChessZero 0.30 Backend: BLAS a b c d 50 100 150 200 250 SE +/- 0.33, N = 3 173 219 225 213 1. (CXX) g++ options: -flto -pthread
LeelaChessZero Backend: Eigen OpenBenchmarking.org Nodes Per Second, More Is Better LeelaChessZero 0.30 Backend: Eigen a b c d 30 60 90 120 150 SE +/- 2.08, N = 3 121 146 154 151 1. (CXX) g++ options: -flto -pthread
Quicksilver Input: CTS2 OpenBenchmarking.org Figure Of Merit, More Is Better Quicksilver 20230818 Input: CTS2 a b c d 4M 8M 12M 16M 20M SE +/- 6666.67, N = 3 20680000 20646667 20620000 20600000 1. (CXX) g++ options: -fopenmp -O3 -march=native
Quicksilver Input: CORAL2 P1 OpenBenchmarking.org Figure Of Merit, More Is Better Quicksilver 20230818 Input: CORAL2 P1 a b c d 5M 10M 15M 20M 25M SE +/- 11547.01, N = 3 24210000 24230000 24240000 24290000 1. (CXX) g++ options: -fopenmp -O3 -march=native
Quicksilver Input: CORAL2 P2 OpenBenchmarking.org Figure Of Merit, More Is Better Quicksilver 20230818 Input: CORAL2 P2 a b c d 5M 10M 15M 20M 25M SE +/- 3333.33, N = 3 24030000 24026667 23890000 23840000 1. (CXX) g++ options: -fopenmp -O3 -march=native
CacheBench Test: Read OpenBenchmarking.org MB/s, More Is Better CacheBench Test: Read a b c d 2K 4K 6K 8K 10K SE +/- 0.09, N = 3 11543.37 11543.16 11543.10 11543.49 MIN: 11542.37 / MAX: 11544.55 MIN: 11542.65 / MAX: 11544.48 MIN: 11542.7 / MAX: 11543.41 MIN: 11542.8 / MAX: 11544.64 1. (CC) gcc options: -O3 -lrt
CacheBench Test: Write OpenBenchmarking.org MB/s, More Is Better CacheBench Test: Write a b c d 15K 30K 45K 60K 75K SE +/- 3.29, N = 3 69134.68 69140.53 69142.44 69142.19 MIN: 68881.15 / MAX: 69208.76 MIN: 68883.98 / MAX: 69225.86 MIN: 68884.8 / MAX: 69218.23 MIN: 68886.61 / MAX: 69217.36 1. (CC) gcc options: -O3 -lrt
CacheBench Test: Read / Modify / Write OpenBenchmarking.org MB/s, More Is Better CacheBench Test: Read / Modify / Write a b c d 30K 60K 90K 120K 150K SE +/- 386.79, N = 3 130857.58 130069.85 130806.25 130851.31 MIN: 112608.55 / MAX: 137126.28 MIN: 101861.72 / MAX: 137133.31 MIN: 112724.52 / MAX: 137125.96 MIN: 112492.8 / MAX: 137124.99 1. (CC) gcc options: -O3 -lrt
LZ4 Compression Compression Level: 1 - Compression Speed OpenBenchmarking.org MB/s, More Is Better LZ4 Compression 1.9.4 Compression Level: 1 - Compression Speed a b c d 200 400 600 800 1000 SE +/- 0.63, N = 3 828.78 829.36 829.15 830.37 1. (CC) gcc options: -O3
LZ4 Compression Compression Level: 1 - Decompression Speed OpenBenchmarking.org MB/s, More Is Better LZ4 Compression 1.9.4 Compression Level: 1 - Decompression Speed a b c d 1100 2200 3300 4400 5500 SE +/- 1.42, N = 3 5019.5 5020.0 5023.2 5019.5 1. (CC) gcc options: -O3
LZ4 Compression Compression Level: 3 - Compression Speed OpenBenchmarking.org MB/s, More Is Better LZ4 Compression 1.9.4 Compression Level: 3 - Compression Speed a b c d 30 60 90 120 150 SE +/- 0.30, N = 3 131.24 131.10 131.40 131.61 1. (CC) gcc options: -O3
LZ4 Compression Compression Level: 3 - Decompression Speed OpenBenchmarking.org MB/s, More Is Better LZ4 Compression 1.9.4 Compression Level: 3 - Decompression Speed a b c d 1000 2000 3000 4000 5000 SE +/- 0.63, N = 3 4595.9 4597.9 4598.0 4596.7 1. (CC) gcc options: -O3
LZ4 Compression Compression Level: 9 - Compression Speed OpenBenchmarking.org MB/s, More Is Better LZ4 Compression 1.9.4 Compression Level: 9 - Compression Speed a b c d 10 20 30 40 50 SE +/- 0.02, N = 3 44.28 44.48 45.49 44.52 1. (CC) gcc options: -O3
LZ4 Compression Compression Level: 9 - Decompression Speed OpenBenchmarking.org MB/s, More Is Better LZ4 Compression 1.9.4 Compression Level: 9 - Decompression Speed a b c d 1000 2000 3000 4000 5000 SE +/- 1.12, N = 3 4840.5 4841.4 4842.4 4844.5 1. (CC) gcc options: -O3
rav1e Speed: 1 OpenBenchmarking.org Frames Per Second, More Is Better rav1e 0.7 Speed: 1 a b c d 0.2372 0.4744 0.7116 0.9488 1.186 SE +/- 0.004, N = 3 1.044 1.048 1.044 1.054
rav1e Speed: 5 OpenBenchmarking.org Frames Per Second, More Is Better rav1e 0.7 Speed: 5 a b c d 0.8755 1.751 2.6265 3.502 4.3775 SE +/- 0.014, N = 3 3.747 3.791 3.769 3.891
rav1e Speed: 6 OpenBenchmarking.org Frames Per Second, More Is Better rav1e 0.7 Speed: 6 a b c d 1.1907 2.3814 3.5721 4.7628 5.9535 SE +/- 0.008, N = 3 5.261 5.292 5.282 5.191
rav1e Speed: 10 OpenBenchmarking.org Frames Per Second, More Is Better rav1e 0.7 Speed: 10 a b c d 3 6 9 12 15 SE +/- 0.11, N = 5 10.63 10.89 10.96 11.02
SVT-AV1 Encoder Mode: Preset 4 - Input: Bosphorus 4K OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 1.8 Encoder Mode: Preset 4 - Input: Bosphorus 4K a b c d 2 4 6 8 10 SE +/- 0.015, N = 3 6.677 6.669 6.678 6.633 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
SVT-AV1 Encoder Mode: Preset 8 - Input: Bosphorus 4K OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 1.8 Encoder Mode: Preset 8 - Input: Bosphorus 4K a b c d 14 28 42 56 70 SE +/- 0.07, N = 3 61.53 61.83 61.47 60.95 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
SVT-AV1 Encoder Mode: Preset 12 - Input: Bosphorus 4K OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 1.8 Encoder Mode: Preset 12 - Input: Bosphorus 4K a b c d 40 80 120 160 200 SE +/- 1.08, N = 3 190.92 190.40 192.16 192.68 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
SVT-AV1 Encoder Mode: Preset 13 - Input: Bosphorus 4K OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 1.8 Encoder Mode: Preset 13 - Input: Bosphorus 4K a b c d 40 80 120 160 200 SE +/- 0.80, N = 3 190.79 192.73 189.25 192.60 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
SVT-AV1 Encoder Mode: Preset 4 - Input: Bosphorus 1080p OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 1.8 Encoder Mode: Preset 4 - Input: Bosphorus 1080p a b c d 5 10 15 20 25 SE +/- 0.08, N = 3 18.80 18.68 18.41 18.92 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
SVT-AV1 Encoder Mode: Preset 8 - Input: Bosphorus 1080p OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 1.8 Encoder Mode: Preset 8 - Input: Bosphorus 1080p a b c d 30 60 90 120 150 SE +/- 0.59, N = 3 122.95 123.29 122.95 122.62 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
SVT-AV1 Encoder Mode: Preset 12 - Input: Bosphorus 1080p OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 1.8 Encoder Mode: Preset 12 - Input: Bosphorus 1080p a b c d 110 220 330 440 550 SE +/- 5.37, N = 5 501.42 506.60 501.16 506.17 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
SVT-AV1 Encoder Mode: Preset 13 - Input: Bosphorus 1080p OpenBenchmarking.org Frames Per Second, More Is Better SVT-AV1 1.8 Encoder Mode: Preset 13 - Input: Bosphorus 1080p a b c d 130 260 390 520 650 SE +/- 7.15, N = 3 543.55 573.04 580.47 565.91 1. (CXX) g++ options: -march=native -mno-avx -mavx2 -mavx512f -mavx512bw -mavx512dq
Y-Cruncher Pi Digits To Calculate: 1B OpenBenchmarking.org Seconds, Fewer Is Better Y-Cruncher 0.8.3 Pi Digits To Calculate: 1B a b c d 4 8 12 16 20 SE +/- 0.01, N = 3 15.55 15.50 15.53 15.50
Y-Cruncher Pi Digits To Calculate: 500M OpenBenchmarking.org Seconds, Fewer Is Better Y-Cruncher 0.8.3 Pi Digits To Calculate: 500M a b c d 2 4 6 8 10 SE +/- 0.007, N = 3 7.325 7.349 7.301 7.297
PyTorch Device: CPU - Batch Size: 1 - Model: ResNet-50 OpenBenchmarking.org batches/sec, More Is Better PyTorch 2.1 Device: CPU - Batch Size: 1 - Model: ResNet-50 a b c d 9 18 27 36 45 SE +/- 0.19, N = 3 40.68 40.42 40.82 40.35 MIN: 37.73 / MAX: 40.91 MIN: 37.5 / MAX: 41 MIN: 37.73 / MAX: 41.05 MIN: 37.34 / MAX: 40.65
PyTorch Device: CPU - Batch Size: 16 - Model: ResNet-50 OpenBenchmarking.org batches/sec, More Is Better PyTorch 2.1 Device: CPU - Batch Size: 16 - Model: ResNet-50 a b c d 8 16 24 32 40 SE +/- 0.12, N = 3 32.50 32.10 31.65 32.21 MIN: 30.56 / MAX: 32.75 MIN: 29.1 / MAX: 32.53 MIN: 29.55 / MAX: 31.86 MIN: 30.29 / MAX: 32.43
PyTorch Device: CPU - Batch Size: 256 - Model: ResNet-50 OpenBenchmarking.org batches/sec, More Is Better PyTorch 2.1 Device: CPU - Batch Size: 256 - Model: ResNet-50 a b c d 7 14 21 28 35 SE +/- 0.11, N = 3 31.92 32.15 31.59 32.07 MIN: 30 / MAX: 32.18 MIN: 30.21 / MAX: 32.69 MIN: 29.73 / MAX: 32.12 MIN: 30.1 / MAX: 32.3
TensorFlow Device: CPU - Batch Size: 1 - Model: VGG-16 OpenBenchmarking.org images/sec, More Is Better TensorFlow 2.12 Device: CPU - Batch Size: 1 - Model: VGG-16 a b c d 0.612 1.224 1.836 2.448 3.06 SE +/- 0.01, N = 3 2.72 2.70 2.72 2.69
TensorFlow Device: CPU - Batch Size: 1 - Model: AlexNet OpenBenchmarking.org images/sec, More Is Better TensorFlow 2.12 Device: CPU - Batch Size: 1 - Model: AlexNet a b c d 2 4 6 8 10 SE +/- 0.01, N = 3 6.26 6.23 6.23 6.21
TensorFlow Device: CPU - Batch Size: 16 - Model: VGG-16 OpenBenchmarking.org images/sec, More Is Better TensorFlow 2.12 Device: CPU - Batch Size: 16 - Model: VGG-16 a b c d 2 4 6 8 10 SE +/- 0.03, N = 3 8.51 8.46 8.48 8.51
TensorFlow Device: CPU - Batch Size: 16 - Model: AlexNet OpenBenchmarking.org images/sec, More Is Better TensorFlow 2.12 Device: CPU - Batch Size: 16 - Model: AlexNet a b c d 20 40 60 80 100 SE +/- 0.20, N = 3 100.44 100.01 100.08 99.90
TensorFlow Device: CPU - Batch Size: 1 - Model: GoogLeNet OpenBenchmarking.org images/sec, More Is Better TensorFlow 2.12 Device: CPU - Batch Size: 1 - Model: GoogLeNet a b c d 4 8 12 16 20 SE +/- 0.09, N = 3 9.94 9.74 16.49 9.73
TensorFlow Device: CPU - Batch Size: 1 - Model: ResNet-50 OpenBenchmarking.org images/sec, More Is Better TensorFlow 2.12 Device: CPU - Batch Size: 1 - Model: ResNet-50 a b c d 2 4 6 8 10 SE +/- 0.05, N = 3 8.85 8.79 8.85 8.89
TensorFlow Device: CPU - Batch Size: 16 - Model: GoogLeNet OpenBenchmarking.org images/sec, More Is Better TensorFlow 2.12 Device: CPU - Batch Size: 16 - Model: GoogLeNet a b c d 14 28 42 56 70 SE +/- 0.36, N = 3 60.85 60.18 60.04 59.82
TensorFlow Device: CPU - Batch Size: 16 - Model: ResNet-50 OpenBenchmarking.org images/sec, More Is Better TensorFlow 2.12 Device: CPU - Batch Size: 16 - Model: ResNet-50 a b c d 5 10 15 20 25 SE +/- 0.08, N = 3 19.87 19.45 19.61 19.81
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream a b c d 6 12 18 24 30 SE +/- 0.05, N = 3 26.84 26.65 26.67 26.69
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream a b c d 100 200 300 400 500 SE +/- 0.27, N = 3 446.63 448.91 449.30 449.16
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream a b c d 5 10 15 20 25 SE +/- 0.01, N = 3 18.39 18.33 18.36 18.40
Neural Magic DeepSparse Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream a b c d 12 24 36 48 60 SE +/- 0.02, N = 3 54.37 54.54 54.46 54.35
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 150 300 450 600 750 SE +/- 0.83, N = 3 685.12 681.00 683.25 681.53
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 4 8 12 16 20 SE +/- 0.02, N = 3 17.50 17.60 17.54 17.59
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 40 80 120 160 200 SE +/- 0.65, N = 3 189.91 193.11 192.98 189.27
Neural Magic DeepSparse Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 1.1881 2.3762 3.5643 4.7524 5.9405 SE +/- 0.0173, N = 3 5.2624 5.1753 5.1787 5.2804
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream a b c d 70 140 210 280 350 SE +/- 0.13, N = 3 307.01 304.77 306.50 305.86
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream a b c d 9 18 27 36 45 SE +/- 0.01, N = 3 39.04 39.33 39.12 39.19
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream a b c d 30 60 90 120 150 SE +/- 0.26, N = 3 157.23 155.48 155.40 155.20
Neural Magic DeepSparse Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream a b c d 2 4 6 8 10 SE +/- 0.0106, N = 3 6.3518 6.4230 6.4266 6.4345
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 400 800 1200 1600 2000 SE +/- 10.49, N = 3 2012.21 1962.70 1970.61 1962.16
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 2 4 6 8 10 SE +/- 0.0329, N = 3 5.9507 6.1010 6.0747 6.1024
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 170 340 510 680 850 SE +/- 1.33, N = 3 757.16 752.78 765.99 757.27
Neural Magic DeepSparse Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 0.2982 0.5964 0.8946 1.1928 1.491 SE +/- 0.0024, N = 3 1.3175 1.3252 1.3025 1.3175
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream a b c d 30 60 90 120 150 SE +/- 0.36, N = 3 150.04 148.24 148.76 149.27
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Asynchronous Multi-Stream a b c d 20 40 60 80 100 SE +/- 0.20, N = 3 79.89 80.85 80.57 80.27
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream a b c d 20 40 60 80 100 SE +/- 0.09, N = 3 98.52 98.56 98.65 98.77
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO - Scenario: Synchronous Single-Stream a b c d 3 6 9 12 15 SE +/- 0.01, N = 3 10.14 10.14 10.13 10.12
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream a b c d 8 16 24 32 40 SE +/- 0.05, N = 3 32.52 32.36 32.42 32.38
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Asynchronous Multi-Stream a b c d 80 160 240 320 400 SE +/- 0.27, N = 3 368.31 369.77 369.62 369.53
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream a b c d 4 8 12 16 20 SE +/- 0.02, N = 3 17.16 17.13 17.24 17.12
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering - Scenario: Synchronous Single-Stream a b c d 13 26 39 52 65 SE +/- 0.07, N = 3 58.25 58.35 58.00 58.41
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream a b c d 70 140 210 280 350 SE +/- 0.76, N = 3 306.99 305.38 306.52 299.83
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream a b c d 9 18 27 36 45 SE +/- 0.09, N = 3 39.05 39.26 39.12 39.98
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream a b c d 30 60 90 120 150 SE +/- 0.08, N = 3 156.97 155.54 155.62 155.19
Neural Magic DeepSparse Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream a b c d 2 4 6 8 10 SE +/- 0.0037, N = 3 6.3619 6.4208 6.4178 6.4346
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 30 60 90 120 150 SE +/- 0.01, N = 3 151.47 149.78 149.95 149.76
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 20 40 60 80 100 SE +/- 0.02, N = 3 79.16 80.05 79.89 80.05
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 20 40 60 80 100 SE +/- 0.29, N = 3 98.91 98.75 98.50 98.89
Neural Magic DeepSparse Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 3 6 9 12 15 SE +/- 0.03, N = 3 10.10 10.12 10.15 10.11
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream a b c d 50 100 150 200 250 SE +/- 0.12, N = 3 224.18 221.71 223.05 222.72
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream a b c d 12 24 36 48 60 SE +/- 0.03, N = 3 53.47 54.06 53.74 53.82
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream a b c d 30 60 90 120 150 SE +/- 0.26, N = 3 112.09 111.29 111.59 111.50
Neural Magic DeepSparse Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream a b c d 3 6 9 12 15 SE +/- 0.0211, N = 3 8.9109 8.9755 8.9520 8.9586
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream a b c d 7 14 21 28 35 SE +/- 0.02, N = 3 30.49 30.28 30.45 30.32
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream a b c d 90 180 270 360 450 SE +/- 0.41, N = 3 392.96 394.78 393.83 394.38
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream a b c d 5 10 15 20 25 SE +/- 0.05, N = 3 21.73 21.47 21.69 21.67
Neural Magic DeepSparse Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream a b c d 11 22 33 44 55 SE +/- 0.12, N = 3 46.00 46.55 46.09 46.13
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 70 140 210 280 350 SE +/- 0.44, N = 3 335.35 333.37 333.64 334.64
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream a b c d 8 16 24 32 40 SE +/- 0.05, N = 3 35.76 35.97 35.93 35.84
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 20 40 60 80 100 SE +/- 0.14, N = 3 76.14 75.51 75.99 75.71
Neural Magic DeepSparse Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream a b c d 3 6 9 12 15 SE +/- 0.02, N = 3 13.13 13.24 13.15 13.20
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream a b c d 6 12 18 24 30 SE +/- 0.03, N = 3 26.91 26.67 26.71 26.54
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream a b c d 100 200 300 400 500 SE +/- 0.32, N = 3 445.26 448.04 448.90 448.03
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream OpenBenchmarking.org items/sec, More Is Better Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream a b c d 5 10 15 20 25 SE +/- 0.01, N = 3 18.47 18.40 18.46 18.44
Neural Magic DeepSparse Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream OpenBenchmarking.org ms/batch, Fewer Is Better Neural Magic DeepSparse 1.6 Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream a b c d 12 24 36 48 60 SE +/- 0.03, N = 3 54.12 54.33 54.15 54.20
Speedb Test: Random Fill OpenBenchmarking.org Op/s, More Is Better Speedb 2.7 Test: Random Fill a b c d 120K 240K 360K 480K 600K SE +/- 4227.88, N = 3 558330 554997 557348 556675 1. (CXX) g++ options: -O3 -march=native -pthread -fno-builtin-memcmp -fno-rtti -lpthread
Speedb Test: Random Read OpenBenchmarking.org Op/s, More Is Better Speedb 2.7 Test: Random Read a b c d 30M 60M 90M 120M 150M SE +/- 81483.06, N = 3 148134848 147432214 146285738 146473036 1. (CXX) g++ options: -O3 -march=native -pthread -fno-builtin-memcmp -fno-rtti -lpthread
Speedb Test: Update Random OpenBenchmarking.org Op/s, More Is Better Speedb 2.7 Test: Update Random a b c d 90K 180K 270K 360K 450K SE +/- 4060.59, N = 3 431692 418848 423788 417457 1. (CXX) g++ options: -O3 -march=native -pthread -fno-builtin-memcmp -fno-rtti -lpthread
Speedb Test: Sequential Fill OpenBenchmarking.org Op/s, More Is Better Speedb 2.7 Test: Sequential Fill a b c d 130K 260K 390K 520K 650K SE +/- 3239.87, N = 3 620607 618776 604758 612428 1. (CXX) g++ options: -O3 -march=native -pthread -fno-builtin-memcmp -fno-rtti -lpthread
Speedb Test: Random Fill Sync OpenBenchmarking.org Op/s, More Is Better Speedb 2.7 Test: Random Fill Sync a b c d 10K 20K 30K 40K 50K SE +/- 66.17, N = 3 47488 47708 47373 47267 1. (CXX) g++ options: -O3 -march=native -pthread -fno-builtin-memcmp -fno-rtti -lpthread
Speedb Test: Read While Writing OpenBenchmarking.org Op/s, More Is Better Speedb 2.7 Test: Read While Writing a b c d 1.5M 3M 4.5M 6M 7.5M SE +/- 60887.57, N = 3 7004007 7070407 7047502 6896007 1. (CXX) g++ options: -O3 -march=native -pthread -fno-builtin-memcmp -fno-rtti -lpthread
Speedb Test: Read Random Write Random OpenBenchmarking.org Op/s, More Is Better Speedb 2.7 Test: Read Random Write Random a b c d 500K 1000K 1500K 2000K 2500K SE +/- 1258.96, N = 3 2327911 2307686 2320670 2316804 1. (CXX) g++ options: -O3 -march=native -pthread -fno-builtin-memcmp -fno-rtti -lpthread
Llama.cpp Model: llama-2-7b.Q4_0.gguf OpenBenchmarking.org Tokens Per Second, More Is Better Llama.cpp b1808 Model: llama-2-7b.Q4_0.gguf a b c d 5 10 15 20 25 SE +/- 0.24, N = 4 20.76 20.95 20.74 20.68 1. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -march=native -mtune=native -lopenblas
Llama.cpp Model: llama-2-13b.Q4_0.gguf OpenBenchmarking.org Tokens Per Second, More Is Better Llama.cpp b1808 Model: llama-2-13b.Q4_0.gguf a b c d 3 6 9 12 15 SE +/- 0.06, N = 3 11.64 11.32 11.27 11.25 1. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -march=native -mtune=native -lopenblas
Llama.cpp Model: llama-2-70b-chat.Q5_0.gguf OpenBenchmarking.org Tokens Per Second, More Is Better Llama.cpp b1808 Model: llama-2-70b-chat.Q5_0.gguf a b c d 0.4388 0.8776 1.3164 1.7552 2.194 SE +/- 0.00, N = 3 1.94 1.94 1.95 1.95 1. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -march=native -mtune=native -lopenblas
Llamafile Test: llava-v1.5-7b-q4 - Acceleration: CPU OpenBenchmarking.org Tokens Per Second, More Is Better Llamafile 0.6 Test: llava-v1.5-7b-q4 - Acceleration: CPU a b c d 4 8 12 16 20 SE +/- 0.01, N = 3 17.22 17.26 17.30 17.25
Llamafile Test: mistral-7b-instruct-v0.2.Q8_0 - Acceleration: CPU OpenBenchmarking.org Tokens Per Second, More Is Better Llamafile 0.6 Test: mistral-7b-instruct-v0.2.Q8_0 - Acceleration: CPU a b c d 3 6 9 12 15 SE +/- 0.01, N = 3 10.13 10.15 10.14 10.13
Llamafile Test: wizardcoder-python-34b-v1.0.Q6_K - Acceleration: CPU OpenBenchmarking.org Tokens Per Second, More Is Better Llamafile 0.6 Test: wizardcoder-python-34b-v1.0.Q6_K - Acceleration: CPU a b c d 0.7313 1.4626 2.1939 2.9252 3.6565 SE +/- 0.00, N = 3 3.25 3.25 3.25 3.25
Phoronix Test Suite v10.8.4