Memtier_benchmark

Memtier_benchmark is a NoSQL Redis/Memcache traffic generation plus benchmarking tool. This current test profile currently just stresses the Redis protocol and basic options exposed wotj a 1:1 Set/Get ratio, 30 pipeline, 100 clients per thread, and thread count equal to the number of CPU cores/threads present. Patches to extend the test are welcome as always. Currently this test profile uses Memtier_benchmark 1.3 and Redis 6.

To run this test with the Phoronix Test Suite, the basic command is: phoronix-test-suite benchmark memtier-benchmark.

Project Site

github.com

Test Created

23 September 2019

Last Updated

10 March 2022

Test Maintainer

Michael Larabel 

Test Type

System

Average Install Time

1 Minute, 18 Seconds

Average Run Time

2 Minutes, 20 Seconds

Test Dependencies

C/C++ Compiler Toolchain + Libevent + Perl Compatible Regular Expressions + Zlib + OpenSSL + Libtool

Accolades

10k+ Downloads

Supported Platforms


Public Result Uploads *Reported Installs **Reported Test Completions **Test Profile Page Views ***OpenBenchmarking.orgEventsMemtier_benchmark Popularity Statisticspts/memtier-benchmark2019.092019.102019.112019.122020.012020.022020.032020.042020.052020.062020.072020.082020.092020.102020.112020.122021.012021.022021.032021.042021.052021.062021.072021.082021.092021.102021.112021.122022.012022.022022.032022.042022.058001600240032004000
* Uploading of benchmark result data to OpenBenchmarking.org is always optional (opt-in) via the Phoronix Test Suite for users wishing to share their results publicly.
** Data based on those opting to upload their test results to OpenBenchmarking.org and users enabling the opt-in anonymous statistics reporting while running benchmarks from an Internet-connected platform.
*** Test profile page view reporting began March 2021.
Data current as of 23 May 2022.

Revision History

pts/memtier-benchmark-1.1.0   [View Source]   Thu, 10 Mar 2022 15:26:52 GMT
Update against upstream Memtier-benchmark 1.3 and Redis 6. Plus add pkg-config as a SystemDependencies. Below patch pointed out pkg-config is needed but it does not work as an ExternalDependencies as pkg-config is not in the XML there, but can be dynamically resolved as SystemDependencies. Closes: https://github.com/phoronix-test-suite/test-profiles/pull/219

pts/memtier-benchmark-1.0.1   [View Source]   Tue, 24 Sep 2019 15:20:16 GMT
Add libtool as possible external dependency.

pts/memtier-benchmark-1.0.0   [View Source]   Mon, 23 Sep 2019 17:41:24 GMT
Initial commit of Redis/memcached memtier_benchmark program.

Suites Using This Test

Database Test Suite

Server


Performance Metrics

Analyze Test Configuration:

Memtier_benchmark 1.3

Protocol: Redis

OpenBenchmarking.org metrics for this test profile configuration based on 97 public results since 10 March 2022 with the latest data as of 28 April 2022.

Below is an overview of the generalized performance for components where there is sufficient statistically significant data based upon user-uploaded results. It is important to keep in mind particularly in the Linux/open-source space there can be vastly different OS configurations, with this overview intended to offer just general guidance as to the performance expectations.

Component
Percentile Rank
# Compatible Public Results
Ops/sec (Average)
97th
5
3772012 +/- 107456
94th
3
3394348 +/- 169020
88th
9
3093655 +/- 193383
78th
5
2796738 +/- 17109
Mid-Tier
75th
< 2781322
74th
3
2714051 +/- 20973
72nd
3
2706703 +/- 94453
58th
4
2245410 +/- 40242
57th
4
2190491 +/- 120137
56th
3
2187233 +/- 27157
Median
50th
2018531
50th
3
1996908 +/- 20848
41st
3
1776684 +/- 12843
40th
3
1758178 +/- 56359
34th
3
1633151 +/- 17006
26th
4
1466378 +/- 87547
Low-Tier
25th
< 1456055
25th
3
1427235 +/- 35341
14th
3
954905 +/- 14624
10th
3
478032 +/- 18179
7th
3
268057 +/- 983
4th
3
261299 +/- 874
OpenBenchmarking.orgDistribution Of Public Results - Protocol: Redis97 Results Range From 260728 To 4437089 Ops/sec260728344256427784511312594840678368761896845424928952101248010960081179536126306413465921430120151364815971761680704176423218477601931288201481620983442181872226540023489282432456251598425995122683040276656828500962933624301715231006803184208326773633512643434792351832036018483685376376890438524323935960401948841030164186544427007243536004437128246810

Based on OpenBenchmarking.org data, the selected test / test configuration (Memtier_benchmark 1.3 - Protocol: Redis) has an average run-time of 7 minutes. By default this test profile is set to run at least 3 times but may increase if the standard deviation exceeds pre-defined defaults or other calculations deem additional runs necessary for greater statistical accuracy of the result.

OpenBenchmarking.orgMinutesTime Required To Complete BenchmarkProtocol: RedisRun-Time510152025Min: 4 / Avg: 6.62 / Max: 22

Based on public OpenBenchmarking.org results, the selected test / test configuration has an average standard deviation of 0.3%.

OpenBenchmarking.orgPercent, Fewer Is BetterAverage Deviation Between RunsProtocol: RedisDeviation246810Min: 0 / Avg: 0.34 / Max: 3

Notable Instruction Set Usage

Notable instruction set extensions supported by this test, based on an automatic analysis by the Phoronix Test Suite / OpenBenchmarking.org analytics engine.

Instruction Set
Support
Instructions Detected
SSE2 (SSE2)
Used by default on supported hardware.
 
CVTSI2SD COMISD UCOMISD MOVAPD DIVSD MULSD CVTTSD2SI ADDSD CVTSS2SD SUBSD MOVDQU MOVDQA MOVD CVTSD2SS SQRTSD
Last automated analysis: 12 March 2022

This test profile binary relies on the shared libraries libevent-2.1.so.7, libcrypto.so.3, libssl.so.3, libm.so.6, libc.so.6.

Tested CPU Architectures

This benchmark has been successfully tested on the below mentioned architectures. The CPU architectures listed is where successful OpenBenchmarking.org result uploads occurred, namely for helping to determine if a given test is compatible with various alternative CPU architectures.

CPU Architecture
Kernel Identifier
Verified On
Intel / AMD x86 64-bit
x86_64
(Many Processors)
IBM Z
s390x
(Many Processors)
ARMv8 64-bit
aarch64
ARMv8 Cortex-A72 4-Core

Recent Test Results

OpenBenchmarking.org Results Compare

1 System - 22 Benchmark Results

AMD Ryzen 9 5950X 16-Core - Gigabyte X570 AORUS ELITE WIFI - AMD Starship

Gentoo - 5.17.4-gentoo-harambe-edition - X Server 1.21.1.3

1 System - 282 Benchmark Results

2 x Intel Xeon Gold 6240 - Intel 440BX - Intel 440BX

CentOS 7.9.2009 - 3.10.0-1160.62.1.el7.x86_64 - X Server

3 Systems - 80 Benchmark Results

AMD Ryzen Threadripper 3960X 24-Core - MSI Creator TRX40 - AMD Starship

Ubuntu 20.04 - 5.12.0-051200rc2daily20210307-generic - GNOME Shell 3.36.4

3 Systems - 93 Benchmark Results

Intel Core i7-8700K - ASUS TUF Z370-PLUS GAMING - Intel 8th Gen Core

Ubuntu 20.04 - 5.9.0-050900rc6daily20200923-generic - GNOME Shell 3.36.4

1 System - 1 Benchmark Result

Intel Core i5-5200U - Dell 01241K - Intel Broadwell-U-OPI

Ubuntu 21.10 - 5.13.0-39-generic - GNOME Shell 40.5

2 Systems - 82 Benchmark Results

AMD Ryzen 9 3950X 16-Core - ASUS ROG CROSSHAIR VII HERO - AMD Starship

Ubuntu 20.04 - 5.11.0-43-generic - GNOME Shell 3.36.4

1 System - 1 Benchmark Result

AMD Ryzen 7 5700G - ASRock X300M-STX - 12GB

Fedora Linux 36 - 5.17.1-300.fc36.x86_64 - GNOME Shell

1 System - 65 Benchmark Results

Unknown - 4096MB - 50GB

Ubuntu 22.04 - 5.15.0-25-generic - 1.2.204

1 System - 56 Benchmark Results

Unknown - 4096MB - 50GB

Ubuntu 22.04 - 5.15.0-25-generic - 1.2.204

1 System - 189 Benchmark Results

2 x Intel Xeon Gold 6252 - Intel 440BX - Intel 440BX

Debian 11 - 5.10.0-13-amd64 - KDE Plasma 5.20.5

11 Systems - 60 Benchmark Results

AMD Ryzen 5 5500 - Gigabyte A520M DS3H - AMD Renoir

Ubuntu 21.10 - 5.16.0-051600rc8-generic - GNOME Shell 40.5

1 System - 1 Benchmark Result

AMD Ryzen 5 5600X 6-Core - ASUS ROG STRIX B550-F GAMING - AMD Starship

openSUSE 20220324 - 5.16.15-1-default - GNOME Shell 42.0

1 System - 1 Benchmark Result

AMD Ryzen 7 1700X Eight-Core - MSI X370 GAMING PRO CARBON - AMD 17h

Ubuntu 20.04 - 5.13.0-37-generic - GNOME Shell 3.36.9

1 System - 291 Benchmark Results

Intel Xeon W-1290P - LENOVO 1048 - Intel Comet Lake PCH

Ubuntu 20.04 - 5.14.0-1027-oem - GNOME Shell 3.36.9

1 System - 1 Benchmark Result

AMD Ryzen 9 5950X 16-Core - ASUS ROG STRIX X570-F GAMING - AMD Starship

Fedora Linux 35 - 5.16.14-200.fc35.x86_64 - GNOME Shell 41.4

Most Popular Test Results

OpenBenchmarking.org Results Compare

3 Systems - 106 Benchmark Results

AMD Ryzen Threadripper 2990WX 32-Core - ASUS ROG ZENITH EXTREME - AMD 17h

Ubuntu 20.10 - 5.8.0-50-generic - GNOME Shell 3.38.1

4 Systems - 39 Benchmark Results

AMD Ryzen 9 3900XT 12-Core - MSI MEG X570 GODLIKE - AMD Starship

Ubuntu 22.04 - 5.15.0-22-generic - GNOME Shell 41.3

3 Systems - 93 Benchmark Results

Intel Core i7-8700K - ASUS TUF Z370-PLUS GAMING - Intel 8th Gen Core

Ubuntu 20.04 - 5.9.0-050900rc6daily20200923-generic - GNOME Shell 3.36.4

3 Systems - 34 Benchmark Results

AMD Ryzen 9 5950X 16-Core - ASRockRack X570D4U-2L2T - 128GB

Ubuntu Jammy Jellyfish - 5.15.0-22-generic - 1.2.195

4 Systems - 38 Benchmark Results

AMD Ryzen 3 3200U - MOTILE PF4PU1F - AMD Raven

Ubuntu 20.04 - 5.13.0-27-generic - GNOME Shell 3.36.9

3 Systems - 23 Benchmark Results

AMD Ryzen 5 4500U - LENOVO LNVNB161216 - AMD Renoir

Ubuntu 21.10 - 5.16.0-051600rc8daily20220108-generic - GNOME Shell 40.5

3 Systems - 42 Benchmark Results

Intel Core i7-10700T - Logic Supply RXM-181 - Intel Comet Lake PCH

Ubuntu 21.10 - 5.13.0-27-generic - GNOME Shell 40.5

2 Systems - 87 Benchmark Results

AMD EPYC 72F3 8-Core - Supermicro H12SSL-i v1.01 - AMD Starship

Ubuntu 21.04 - 5.17.0-051700rc7daily20220310-generic - GNOME Shell 3.38.4

2 Systems - 82 Benchmark Results

AMD Ryzen 9 3950X 16-Core - ASUS ROG CROSSHAIR VII HERO - AMD Starship

Ubuntu 20.04 - 5.11.0-43-generic - GNOME Shell 3.36.4

3 Systems - 80 Benchmark Results

AMD Ryzen Threadripper 3960X 24-Core - MSI Creator TRX40 - AMD Starship

Ubuntu 20.04 - 5.12.0-051200rc2daily20210307-generic - GNOME Shell 3.36.4

3 Systems - 23 Benchmark Results

Intel Core i9-10900K - Gigabyte Z490 AORUS MASTER - Intel Comet Lake PCH

Ubuntu 21.10 - 5.17.0-051700rc7daily20220309-generic - GNOME Shell 40.5

1 System - 291 Benchmark Results

Intel Xeon W-1290P - LENOVO 1048 - Intel Comet Lake PCH

Ubuntu 20.04 - 5.14.0-1027-oem - GNOME Shell 3.36.9

3 Systems - 34 Benchmark Results

Intel Xeon Silver 4216 - TYAN S7100AG2NR - Intel Sky Lake-E DMI3 Registers

Debian 11 - 5.10.0-10-amd64 - X Server

3 Systems - 23 Benchmark Results

AMD Ryzen 9 5950X 16-Core - ASUS ROG CROSSHAIR VIII HERO - AMD Starship

Ubuntu 20.04 - 5.17.0-051700rc7daily20220310-generic - GNOME Shell 3.36.9

11 Systems - 60 Benchmark Results

AMD Ryzen 5 5500 - Gigabyte A520M DS3H - AMD Renoir

Ubuntu 21.10 - 5.16.0-051600rc8-generic - GNOME Shell 40.5

Find More Test Results