hw-mig AMD EPYC 7R13 48-Core testing with a Supermicro H12SSL-I v1.02 (2.7 BIOS) and NVIDIA GeForce RTX 4090 24GB on EndeavourOS rolling via the Phoronix Test Suite.
HTML result view exported from: https://openbenchmarking.org/result/2402101-NE-HWMIG553265 .
hw-mig Processor Motherboard Chipset Memory Disk Graphics Audio Monitor Network OS Kernel Desktop Display Server Display Driver OpenGL Compiler File-System Screen Resolution AMD EPYC 7R13 48-Core - NVIDIA GeForce RTX 4090 24GB AMD EPYC 7R13 48-Core @ 3.73GHz (48 Cores / 96 Threads) Supermicro H12SSL-I v1.02 (2.7 BIOS) AMD Starship/Matisse 256GB 15363GB Micron_7450_MTFDKCC15T3TFR NVIDIA GeForce RTX 4090 24GB NVIDIA AD102 HD Audio 38GN950 2 x Intel X710 for 10GbE SFP+ EndeavourOS rolling 6.7.4-zen1-1-zen (x86_64) Xfce 4.18 X Server 1.21.1.11 NVIDIA 545.29.06 4.6.0 GCC 13.2.1 20230801 + Clang 16.0.6 + LLVM 16.0.6 + CUDA 12.3 btrfs 3840x1600 OpenBenchmarking.org - Transparent Huge Pages: always - NVCC_PREPEND_FLAGS="-ccbin /opt/cuda/bin" - --disable-libssp --disable-libstdcxx-pch --disable-werror --enable-__cxa_atexit --enable-bootstrap --enable-cet=auto --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-default-ssp --enable-gnu-indirect-function --enable-gnu-unique-object --enable-languages=ada,c,c++,d,fortran,go,lto,m2,objc,obj-c++ --enable-libstdcxx-backtrace --enable-link-serialization=1 --enable-lto --enable-multilib --enable-plugin --enable-shared --enable-threads=posix --mandir=/usr/share/man --with-build-config=bootstrap-lto --with-linker-hash-style=gnu - Scaling Governor: amd-pstate-epp performance (EPP: performance) - CPU Microcode: 0xa0011d1 - gather_data_sampling: Not affected + itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Not affected + spec_rstack_overflow: Mitigation of Safe RET + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional IBRS_FW STIBP: always-on RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected
hw-mig llama-cpp: llama-2-7b.Q4_0.gguf llama-cpp: llama-2-13b.Q4_0.gguf llama-cpp: llama-2-70b-chat.Q5_0.gguf llamafile: llava-v1.5-7b-q4 - CPU llamafile: mistral-7b-instruct-v0.2.Q8_0 - CPU llamafile: wizardcoder-python-34b-v1.0.Q6_K - CPU AMD EPYC 7R13 48-Core - NVIDIA GeForce RTX 4090 24GB 23.17 14.45 2.86 22.30 13.34 4.73 OpenBenchmarking.org
Llama.cpp Model: llama-2-7b.Q4_0.gguf OpenBenchmarking.org Tokens Per Second, More Is Better Llama.cpp b1808 Model: llama-2-7b.Q4_0.gguf AMD EPYC 7R13 48-Core - NVIDIA GeForce RTX 4090 24GB 6 12 18 24 30 SE +/- 0.03, N = 3 23.17 1. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -march=native -mtune=native -fopenmp -lopenblas
Llama.cpp Model: llama-2-13b.Q4_0.gguf OpenBenchmarking.org Tokens Per Second, More Is Better Llama.cpp b1808 Model: llama-2-13b.Q4_0.gguf AMD EPYC 7R13 48-Core - NVIDIA GeForce RTX 4090 24GB 4 8 12 16 20 SE +/- 0.01, N = 3 14.45 1. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -march=native -mtune=native -fopenmp -lopenblas
Llama.cpp Model: llama-2-70b-chat.Q5_0.gguf OpenBenchmarking.org Tokens Per Second, More Is Better Llama.cpp b1808 Model: llama-2-70b-chat.Q5_0.gguf AMD EPYC 7R13 48-Core - NVIDIA GeForce RTX 4090 24GB 0.6435 1.287 1.9305 2.574 3.2175 SE +/- 0.01, N = 3 2.86 1. (CXX) g++ options: -std=c++11 -fPIC -O3 -pthread -march=native -mtune=native -fopenmp -lopenblas
Llamafile Test: llava-v1.5-7b-q4 - Acceleration: CPU OpenBenchmarking.org Tokens Per Second, More Is Better Llamafile 0.6 Test: llava-v1.5-7b-q4 - Acceleration: CPU AMD EPYC 7R13 48-Core - NVIDIA GeForce RTX 4090 24GB 5 10 15 20 25 SE +/- 0.05, N = 3 22.30
Llamafile Test: mistral-7b-instruct-v0.2.Q8_0 - Acceleration: CPU OpenBenchmarking.org Tokens Per Second, More Is Better Llamafile 0.6 Test: mistral-7b-instruct-v0.2.Q8_0 - Acceleration: CPU AMD EPYC 7R13 48-Core - NVIDIA GeForce RTX 4090 24GB 3 6 9 12 15 SE +/- 0.03, N = 3 13.34
Llamafile Test: wizardcoder-python-34b-v1.0.Q6_K - Acceleration: CPU OpenBenchmarking.org Tokens Per Second, More Is Better Llamafile 0.6 Test: wizardcoder-python-34b-v1.0.Q6_K - Acceleration: CPU AMD EPYC 7R13 48-Core - NVIDIA GeForce RTX 4090 24GB 1.0643 2.1286 3.1929 4.2572 5.3215 SE +/- 0.01, N = 3 4.73
Phoronix Test Suite v10.8.4