Memtier_benchmark

Memtier_benchmark is a NoSQL Redis/Memcache traffic generation plus benchmarking tool. This current test profile currently just stresses the Redis protocol and basic options exposed wotj a 1:1 Set/Get ratio, 30 pipeline, 100 clients per thread, and thread count equal to the number of CPU cores/threads present. Patches to extend the test are welcome as always.

To run this test with the Phoronix Test Suite, the basic command is: phoronix-test-suite benchmark memtier-benchmark.

Project Site

github.com

Test Created

23 September 2019

Last Updated

24 September 2019

Test Maintainer

Michael Larabel 

Test Type

System

Average Install Time

1 Minute, 10 Seconds

Average Run Time

2 Minutes, 20 Seconds

Test Dependencies

C/C++ Compiler Toolchain + Libevent + Perl Compatible Regular Expressions + OpenSSL + Libtool

Accolades

10k+ Downloads

Supported Platforms


Public Result Uploads *Reported Installs **Reported Test Completions **Test Profile Page Views ***OpenBenchmarking.orgEventsMemtier_benchmark Popularity Statisticspts/memtier-benchmark2019.092019.102019.112019.122020.012020.022020.032020.042020.052020.062020.072020.082020.092020.102020.112020.122021.012021.022021.032021.042021.052021.062021.072021.082021.097001400210028003500
* Uploading of benchmark result data to OpenBenchmarking.org is always optional (opt-in) via the Phoronix Test Suite for users wishing to share their results publicly.
** Data based on those opting to upload their test results to OpenBenchmarking.org and users enabling the opt-in anonymous statistics reporting while running benchmarks from an Internet-connected platform.
*** Test profile page view reporting began March 2021.
Data current as of 24 September 2021.

Revision History

pts/memtier-benchmark-1.0.1   [View Source]   Tue, 24 Sep 2019 15:20:16 GMT
Add libtool as possible external dependency.

pts/memtier-benchmark-1.0.0   [View Source]   Mon, 23 Sep 2019 17:41:24 GMT
Initial commit of Redis/memcached memtier_benchmark program.

Suites Using This Test

Database Test Suite

Server


Performance Metrics

Analyze Test Configuration:

Memtier_benchmark 1.2.17

Protocol: Redis

OpenBenchmarking.org metrics for this test profile configuration based on 257 public results since 23 September 2019 with the latest data as of 23 August 2021.

Below is an overview of the generalized performance for components where there is sufficient statistically significant data based upon user-uploaded results. It is important to keep in mind particularly in the Linux/open-source space there can be vastly different OS configurations, with this overview intended to offer just general guidance as to the performance expectations.

Component
Percentile Rank
# Compatible Public Results
Ops/sec (Average)
98th
22
4145403 +/- 338761
93rd
3
3715581 +/- 187539
87th
3
2890309 +/- 377632
86th
4
2885454 +/- 69442
83rd
4
2792857 +/- 102621
82nd
10
2784259 +/- 113870
79th
7
2743850 +/- 113220
Mid-Tier
75th
< 2672147
75th
6
2653786 +/- 121145
68th
27
2562052 +/- 136376
56th
4
2361418 +/- 38584
51st
6
2285680 +/- 42825
Median
50th
2272891
42nd
5
2044125 +/- 49334
40th
3
1997000 +/- 71726
37th
3
1950393 +/- 29150
32nd
3
1874687 +/- 6438
30th
6
1808923 +/- 170232
26th
3
1645381 +/- 145270
Low-Tier
25th
< 1623975
24th
7
1596694 +/- 15939
12th
10
1212755 +/- 76940
7th
9
1114351 +/- 19115
OpenBenchmarking.orgDistribution Of Public Results - Protocol: Redis257 Results Range From 176928 To 4800891 Ops/sec176928269408361888454368546848639328731808824288916768100924811017281194208128668813791681471648156412816566081749088184156819340482026528211900822114882303968239644824889282581408267388827663682858848295132830438083136288322876833212483413728350620835986883691168378364838761283968608406108841535684246048433852844310084523488461596847084484800928510152025

Based on OpenBenchmarking.org data, the selected test / test configuration (Memtier_benchmark 1.2.17 - Protocol: Redis) has an average run-time of 10 minutes. By default this test profile is set to run at least 3 times but may increase if the standard deviation exceeds pre-defined defaults or other calculations deem additional runs necessary for greater statistical accuracy of the result.

OpenBenchmarking.orgMinutesTime Required To Complete BenchmarkProtocol: RedisRun-Time816243240Min: 4 / Avg: 10.14 / Max: 40

Based on public OpenBenchmarking.org results, the selected test / test configuration has an average standard deviation of 2.3%.

OpenBenchmarking.orgPercent, Fewer Is BetterAverage Deviation Between RunsProtocol: RedisDeviation3691215Min: 0 / Avg: 2.3 / Max: 11

Notable Instruction Set Usage

Notable instruction set extensions supported by this test, based on an automatic analysis by the Phoronix Test Suite / OpenBenchmarking.org analytics engine.

Instruction Set
Support
Instructions Detected
SSE2 (SSE2)
Used by default on supported hardware.
 
CVTSI2SD COMISD UCOMISD MOVAPD DIVSD MULSD CVTTSD2SI ADDSD CVTSS2SD SUBSD MOVDQU MOVD CVTSD2SS SQRTSD MOVDQA
Last automated analysis: 10 May 2021

This test profile binary relies on the shared libraries libevent-2.1.so.7, libpthread.so.0, libm.so.6, libc.so.6.

Recent Test Results

OpenBenchmarking.org Results Compare

1 System - 127 Benchmark Results

AMD EPYC 7302P 16-Core - HPE ProLiant DL325 Gen10 Plus - AMD Starship

Ubuntu 20.04 - 5.4.0-81-generic - GNOME Shell

1 System - 106 Benchmark Results

2 x Intel Xeon E5-2680 0 - Cisco UCSC-C220-M3S - Intel Xeon E5

CentOS 7.9.2009 - 3.10.0-1160.36.2.el7.x86_64 - X Server

1 System - 99 Benchmark Results

2 x Intel Xeon E5-2680 0 - Cisco UCSC-C220-M3S - Intel Xeon E5

CentOS 7.9.2009 - 3.10.0-1160.11.1.el7.x86_64 - X Server

1 System - 100 Benchmark Results

2 x Intel Xeon E5-2680 v3 - Intel 440BX - Intel 440BX

CentOS 7.9.2009 - 3.10.0-1160.36.2.el7.x86_64 - X Server

1 System - 106 Benchmark Results

2 x Intel Xeon E5-2680 v3 - Intel 440BX - Intel 440BX

CentOS 7.9.2009 - 3.10.0-1160.11.1.el7.x86_64 - X Server

1 System - 100 Benchmark Results

2 x Intel Xeon E5-2680 v3 - Intel 440BX - Intel 440BX

CentOS 7.9.2009 - 3.10.0-1160.36.2.el7.x86_64 - X Server

1 System - 106 Benchmark Results

2 x Intel Xeon E5-2680 0 - Cisco UCSC-C220-M3S - Intel Xeon E5

CentOS 7.9.2009 - 3.10.0-1160.36.2.el7.x86_64 - X Server

1 System - 99 Benchmark Results

2 x Intel Xeon E5-2680 0 - Cisco UCSC-C220-M3S - Intel Xeon E5

CentOS 7.9.2009 - 3.10.0-1160.11.1.el7.x86_64 - X Server

1 System - 85 Benchmark Results

2 x Intel Xeon E5-2680 0 - Cisco UCSC-C220-M3S - Intel Xeon E5

CentOS 7.9.2009 - 3.10.0-1160.36.2.el7.x86_64 - X Server

1 System - 106 Benchmark Results

2 x Intel Xeon E5-2680 v3 - Intel 440BX - Intel 440BX

CentOS 7.9.2009 - 3.10.0-1160.11.1.el7.x86_64 - X Server

1 System - 206 Benchmark Results

AMD EPYC 7352 24-Core - Dell 07PXPY - AMD Starship

Debian 10 - 4.19.0-17-amd64 - GCC 8.3.0

1 System - 125 Benchmark Results

4 x Intel Xeon E5-4650 0 - Dell 04K5X5 - Intel Xeon E5

CentOS Linux 8 - 4.18.0-305.3.1.el8.x86_64 - GNOME Shell 3.32.2

1 System - 116 Benchmark Results

4 x Intel Xeon E5-4650 0 - Dell 04K5X5 - Intel Xeon E5

CentOS Linux 8 - 4.18.0-305.3.1.el8.x86_64 - GNOME Shell 3.32.2

1 System - 176 Benchmark Results

AMD EPYC 7B12 - Google Compute Engine n2d-standard-16 - Intel 440FX 82441FX PMC

Ubuntu 20.04 - 5.4.0-1040-gcp - GCC 9.3.0

1 System - 171 Benchmark Results

2 x Intel Xeon Gold 5218R - Supermicro X11DPX-T - Intel Device 2020

Ubuntu 16.04 - 4.15.0-139-generic - Unity 7.4.0

Most Popular Test Results

OpenBenchmarking.org Results Compare

2 Systems - 403 Benchmark Results

Intel Core i9-10900K - Gigabyte Z490 AORUS MASTER - Intel Comet Lake PCH

Ubuntu 20.10 - 5.8.0-22-generic - GNOME Shell 3.38.0

3 Systems - 406 Benchmark Results

Intel Core i9-10900K - Gigabyte Z490 AORUS MASTER - Intel Comet Lake PCH

Ubuntu 20.04 - 5.4.0-48-generic - GNOME Shell 3.36.4

2 Systems - 475 Benchmark Results

AMD Ryzen Threadripper 3970X 32-Core - ASUS ROG ZENITH II EXTREME - AMD Starship

Ubuntu 19.10 - 5.3.0-18-generic - GNOME Shell 3.34.1

1 System - 17 Benchmark Results

Intel Core i5-8400 - MSI Z370M MORTAR - Intel 8th Gen Core

Ubuntu 19.04 - 5.0.0-29-generic - GNOME Shell 3.32.1

3 Systems - 25 Benchmark Results

2 x Intel Core - QEMU Standard PC - Intel 440FX 82441FX PMC

Ubuntu 20.04 - 5.4.0-52-generic - GCC 9.3.0

2 Systems - 345 Benchmark Results

AMD Ryzen Threadripper 3960X 24-Core - MSI Creator TRX40 - AMD Starship

Ubuntu 19.10 - 5.3.0-23-generic - GNOME Shell 3.34.1

2 Systems - 406 Benchmark Results

AMD Ryzen 9 3900XT 12-Core - MSI MEG X570 GODLIKE - AMD Starship

Ubuntu 20.10 - 5.8.0-20-generic - GNOME Shell 3.38.0

2 Systems - 59 Benchmark Results

2 x AMD EPYC 7402 24-Core - Dell 0GK70M - 8 x 64 GB DDR4-3200MT

CentOS Linux 8 - 4.18.0-193.28.1.el8_2.x86_64 - GCC 8.3.1 20191121

2 Systems - 35 Benchmark Results

4 x AMD Opteron 62xx class - QEMU Standard PC - Intel 440FX 82441FX PMC

Ubuntu 20.04 - 5.4.0-52-generic - GCC 9.3.0

1 System - 10 Benchmark Results

AMD Ryzen 9 3900X 12-Core - ASUS ROG CROSSHAIR VIII HERO - AMD Starship

Ubuntu 19.10 - 5.3.0-10-generic - GNOME Shell 3.34.0

3 Systems - 335 Benchmark Results

AMD Ryzen Threadripper 3960X 24-Core - MSI Creator TRX40 - AMD Starship

Ubuntu 19.10 - 5.3.0-23-generic - GNOME Shell 3.34.1

6 Systems - 116 Benchmark Results

Intel Core i9-10980XE - Gigabyte X299X DESIGNARE 10G - Intel Sky Lake-E DMI3 Registers

CentOS Linux 8 - 4.18.0-80.11.2.el8_0.x86_64 - GNOME Shell 3.28.3

Find More Test Results