Memtier_benchmark

Memtier_benchmark is a NoSQL Redis/Memcache traffic generation plus benchmarking tool. This current test profile currently just stresses the Redis protocol and basic options exposed wotj a 1:1 Set/Get ratio, 30 pipeline, 100 clients per thread, and thread count equal to the number of CPU cores/threads present. Patches to extend the test are welcome as always.

To run this test with the Phoronix Test Suite, the basic command is: phoronix-test-suite benchmark memtier-benchmark.

Project Site

github.com

Test Created

23 September 2019

Last Updated

24 September 2019

Test Maintainer

Michael Larabel 

Test Type

System

Average Install Time

1 Minute, 11 Seconds

Average Run Time

2 Minutes, 20 Seconds

Test Dependencies

C/C++ Compiler Toolchain + Libevent + Perl Compatible Regular Expressions + OpenSSL + Libtool

Accolades

10k+ Downloads

Supported Platforms


Public Result UploadsReported Installs*Test Completions*OpenBenchmarking.orgEventsMemtier_benchmark Popularity Statisticspts/memtier-benchmark2019.092019.102019.112019.122020.012020.022020.032020.042020.052020.062020.072020.082020.092020.102020.112020.122021.012021.022021.035001000150020002500
* Data based on those opting to upload their test results to OpenBenchmarking.org and users enabling the opt-in anonymous statistics reporting while running benchmarks from an Internet-connected platform.
Data current as of Thu, 04 Mar 2021 14:20:40 GMT.

Revision History

pts/memtier-benchmark-1.0.1   [View Source]   Tue, 24 Sep 2019 15:20:16 GMT
Add libtool as possible external dependency.

pts/memtier-benchmark-1.0.0   [View Source]   Mon, 23 Sep 2019 17:41:24 GMT
Initial commit of Redis/memcached memtier_benchmark program.

Suites Using This Test

Database Test Suite

Server


Performance Metrics

Analyze Test Configuration:

Memtier_benchmark 1.2.17

Protocol: Redis

OpenBenchmarking.org metrics for this test profile configuration based on 242 public results since 23 September 2019 with the latest data as of 24 February 2021.

Below is an overview of the generalized performance for components where there is sufficient statistically significant data based upon user-uploaded results. It is important to keep in mind particularly in the Linux/open-source space there can be vastly different OS configurations, with this overview intended to offer just general guidance as to the performance expectations.

Component
Percentile Rank
# Matching Public Results
Ops/sec (Average)
98th
22
4145403 +/- 338761
93rd
3
3715581 +/- 187539
86th
3
2890309 +/- 377632
85th
4
2885454 +/- 69442
82nd
4
2792857 +/- 102621
82nd
10
2784259 +/- 113870
78th
7
2743850 +/- 113220
Mid-Tier
75th
< 2679836
73rd
6
2653786 +/- 121145
67th
27
2562052 +/- 136376
54th
4
2361418 +/- 38584
Median
50th
2306793
48th
6
2285680 +/- 42825
39th
5
2044125 +/- 49334
37th
3
1997000 +/- 71726
34th
3
1950393 +/- 29150
28th
3
1874687 +/- 6438
27th
5
1866427 +/- 106880
Low-Tier
25th
< 1806239
21st
7
1596694 +/- 15939
11th
10
1212755 +/- 76940
7th
4
1129840 +/- 15236
OpenBenchmarking.orgDistribution Of Public Results - Protocol: Redis242 Results Range From 176928 To 4800891 Ops/sec176928269408361888454368546848639328731808824288916768100924811017281194208128668813791681471648156412816566081749088184156819340482026528211900822114882303968239644824889282581408267388827663682858848295132830438083136288322876833212483413728350620835986883691168378364838761283968608406108841535684246048433852844310084523488461596847084484800928510152025

Based on OpenBenchmarking.org data, the selected test / test configuration (Memtier_benchmark 1.2.17 - Protocol: Redis) has an average run-time of 10 minutes. By default this test profile is set to run at least 3 times but may increase if the standard deviation exceeds pre-defined defaults or other calculations deem additional runs necessary for greater statistical accuracy of the result.

OpenBenchmarking.orgMinutesTime Required To Complete BenchmarkProtocol: RedisRun-Time816243240Min: 4 / Avg: 10.13 / Max: 40

Based on public OpenBenchmarking.org results, the selected test / test configuration has an average standard deviation of 2.3%.

OpenBenchmarking.orgPercent, Fewer Is BetterAverage Deviation Between RunsProtocol: RedisDeviation3691215Min: 0 / Avg: 2.34 / Max: 11

Notable Instruction Set Usage

Notable instruction set extensions supported by this test, based on an automatic analysis by the Phoronix Test Suite / OpenBenchmarking.org analytics engine.

Instruction Set
Support
Instructions Detected
SSE2 (SSE2)
Used by default on supported hardware.
 
CVTSI2SD COMISD UCOMISD MOVAPD DIVSD MULSD CVTTSD2SI ADDSD CVTSS2SD SUBSD MOVDQU MOVDQA MOVD CVTSD2SS SQRTSD
Last automated analysis: 30 January 2021

This test profile binary relies on the shared libraries libevent-2.1.so.7, libpthread.so.0, libz.so.1, libpcre.so.3, libm.so.6, libc.so.6.

Recent Test Results

OpenBenchmarking.org Results Compare

1 System - 140 Benchmark Results

2 x Intel Xeon E5-2680 v3 - Cisco UCSB-B200-M4 - Intel Xeon E7 v3

OracleServer 8.3 - 5.4.17-2011.7.4.el8uek.x86_64 - matrox

1 System - 117 Benchmark Results

2 x Intel Xeon E5-2680 0 - Cisco UCSC-C220-M3S - Intel Xeon E5

CentOS 7.9.2009 - 3.10.0-1160.11.1.el7.x86_64 - X Server

1 System - 91 Benchmark Results

2 x Intel Xeon E5-2680 0 - Cisco UCSC-C220-M3S - Intel Xeon E5

CentOS 7.4.1708 - 3.10.0-693.2.2.el7.x86_64 - X Server

1 System - 117 Benchmark Results

2 x Intel Xeon E5-2680 0 - Cisco UCSC-C220-M3S - Intel Xeon E5

CentOS 7.4.1708 - 3.10.0-693.2.2.el7.x86_64 - X Server

1 System - 94 Benchmark Results

2 x Intel Xeon Gold 6240 - Intel 440BX - Intel 440BX

Amazon Linux 2 - 4.14.214-160.339.amzn2.x86_64 - vmware

1 System - 117 Benchmark Results

2 x Intel Xeon E5-2680 v3 - Cisco UCSB-B200-M4 - Intel Xeon E7 v3

OracleServer 7.4 - 4.1.12-103.7.3.el7uek.x86_64 - matrox

1 System - 140 Benchmark Results

2 x Intel Xeon E5-2680 v3 - Cisco UCSB-B200-M4 - Intel Xeon E7 v3

OracleServer 8.3 - 5.4.17-2011.7.4.el8uek.x86_64 - matrox

1 System - 117 Benchmark Results

2 x Intel Xeon E5-2680 0 - Cisco UCSC-C220-M3S - Intel Xeon E5

CentOS 7.9.2009 - 3.10.0-1160.11.1.el7.x86_64 - X Server

3 Systems - 25 Benchmark Results

2 x Intel Core - QEMU Standard PC - Intel 440FX 82441FX PMC

Ubuntu 20.04 - 5.4.0-52-generic - GCC 9.3.0

2 Systems - 35 Benchmark Results

4 x AMD Opteron 62xx class - QEMU Standard PC - Intel 440FX 82441FX PMC

Ubuntu 20.04 - 5.4.0-52-generic - GCC 9.3.0

1 System - 19 Benchmark Results

4 x AMD Opteron 62xx class - QEMU Standard PC - Intel 440FX 82441FX PMC

Ubuntu 20.04 - 5.4.0-52-generic - GCC 9.3.0

2 Systems - 17 Benchmark Results

2 x Intel Core - QEMU Standard PC - Intel 440FX 82441FX PMC

Ubuntu 20.04 - 5.4.0-52-generic - GCC 9.3.0

Most Popular Test Results

OpenBenchmarking.org Results Compare

2 Systems - 403 Benchmark Results

Intel Core i9-10900K - Gigabyte Z490 AORUS MASTER - Intel Comet Lake PCH

Ubuntu 20.04 - 5.4.0-48-generic - GNOME Shell 3.36.4

3 Systems - 406 Benchmark Results

AMD Ryzen 9 3900XT 12-Core - MSI MEG X570 GODLIKE - AMD Starship

Ubuntu 20.04 - 5.4.0-48-generic - GNOME Shell 3.36.4

2 Systems - 475 Benchmark Results

AMD Ryzen Threadripper 3970X 32-Core - ASUS ROG ZENITH II EXTREME - AMD Starship

Ubuntu 19.10 - 5.3.0-40-generic - GNOME Shell 3.34.1

2 Systems - 345 Benchmark Results

AMD Ryzen Threadripper 3960X 24-Core - MSI Creator TRX40 - AMD Starship

Ubuntu 19.10 - 5.3.0-23-generic - GNOME Shell 3.34.1

1 System - 376 Benchmark Results

AMD Ryzen 9 3900X 12-Core - ASUS ROG CROSSHAIR VIII HERO - AMD Starship

Ubuntu 19.10 - 5.3.0-10-generic - GNOME Shell 3.34.0

1 System - 10 Benchmark Results

AMD Ryzen 9 3900X 12-Core - ASUS ROG CROSSHAIR VIII HERO - AMD Starship

Ubuntu 19.10 - 5.3.0-10-generic - GNOME Shell 3.34.0

1 System - 17 Benchmark Results

Intel Core i5-8400 - MSI Z370M MORTAR - Intel 8th Gen Core

Ubuntu 19.04 - 5.0.0-29-generic - GNOME Shell 3.32.1

3 Systems - 335 Benchmark Results

AMD Ryzen Threadripper 3960X 24-Core - MSI Creator TRX40 - AMD Starship

Ubuntu 19.10 - 5.3.0-23-generic - GNOME Shell 3.34.1

3 Systems - 25 Benchmark Results

2 x Intel Core - QEMU Standard PC - Intel 440FX 82441FX PMC

Ubuntu 20.04 - 5.4.0-52-generic - GCC 9.3.0

2 Systems - 406 Benchmark Results

AMD Ryzen 9 3900XT 12-Core - MSI MEG X570 GODLIKE - AMD Starship

Ubuntu 20.04 - 5.4.0-48-generic - GNOME Shell 3.36.4

Find More Test Results