memtier_benchmark

Memtier_benchmark is a NoSQL Redis/Memcache traffic generation plus benchmarking tool developed by Redis Labs.

To run this test with the Phoronix Test Suite, the basic command is: phoronix-test-suite benchmark memtier-benchmark.

Project Site

github.com

Source Repository

github.com

Test Created

23 September 2019

Last Updated

19 August 2022

Test Maintainer

Michael Larabel 

Test Type

System

Average Install Time

1 Minute, 30 Seconds

Average Run Time

2 Minutes, 17 Seconds

Test Dependencies

C/C++ Compiler Toolchain + Libevent + Perl Compatible Regular Expressions + Zlib + OpenSSL + Libtool

Accolades

20k+ Downloads

Supported Platforms


Public Result Uploads *Reported Installs **Reported Test Completions **Test Profile Page Views ***OpenBenchmarking.orgEventsMemtier_benchmark Popularity Statisticspts/memtier-benchmark2019.092019.102019.112019.122020.012020.022020.032020.042020.052020.062020.072020.082020.092020.102020.112020.122021.012021.022021.032021.042021.052021.062021.072021.082021.092021.102021.112021.122022.012022.022022.032022.042022.052022.062022.072022.082022.0912002400360048006000
* Uploading of benchmark result data to OpenBenchmarking.org is always optional (opt-in) via the Phoronix Test Suite for users wishing to share their results publicly.
** Data based on those opting to upload their test results to OpenBenchmarking.org and users enabling the opt-in anonymous statistics reporting while running benchmarks from an Internet-connected platform.
*** Test profile page view reporting began March 2021.
Data current as of 28 September 2022.
5040.1%50023.2%10036.7%Clients Option PopularityOpenBenchmarking.org
1:124.8%1:1025.2%5:125.0%1:525.0%Set To Get Ratio Option PopularityOpenBenchmarking.org

Revision History

pts/memtier-benchmark-1.4.0   [View Source]   Fri, 19 Aug 2022 07:37:52 GMT
Update against memtier-benchmark 1.4, Redis 7.0.4.

pts/memtier-benchmark-1.1.0   [View Source]   Thu, 10 Mar 2022 15:26:52 GMT
Update against upstream Memtier-benchmark 1.3 and Redis 6. Plus add pkg-config as a SystemDependencies. Below patch pointed out pkg-config is needed but it does not work as an ExternalDependencies as pkg-config is not in the XML there, but can be dynamically resolved as SystemDependencies. Closes: https://github.com/phoronix-test-suite/test-profiles/pull/219

pts/memtier-benchmark-1.0.1   [View Source]   Tue, 24 Sep 2019 15:20:16 GMT
Add libtool as possible external dependency.

pts/memtier-benchmark-1.0.0   [View Source]   Mon, 23 Sep 2019 17:41:24 GMT
Initial commit of Redis/memcached memtier_benchmark program.

Suites Using This Test

Database Test Suite

Server


Performance Metrics

Analyze Test Configuration:

memtier_benchmark 1.4

Protocol: Redis - Clients: 50 - Set To Get Ratio: 1:10

OpenBenchmarking.org metrics for this test profile configuration based on 163 public results since 19 August 2022 with the latest data as of 27 September 2022.

Below is an overview of the generalized performance for components where there is sufficient statistically significant data based upon user-uploaded results. It is important to keep in mind particularly in the Linux/open-source space there can be vastly different OS configurations, with this overview intended to offer just general guidance as to the performance expectations.

Component
Percentile Rank
# Compatible Public Results
Ops/sec (Average)
98th
4
2969224 +/- 204842
97th
3
2815139 +/- 46305
94th
4
2581846 +/- 194839
94th
4
2572921 +/- 53190
89th
3
2454971 +/- 63308
87th
3
2404420 +/- 22354
84th
3
2382299 +/- 51915
83rd
4
2369593 +/- 8781
Mid-Tier
75th
< 1947715
74th
3
1931376 +/- 27591
74th
3
1929010 +/- 50383
73rd
3
1914116 +/- 40930
71st
4
1903140 +/- 26165
62nd
4
1763695 +/- 38592
57th
4
1733808 +/- 64674
55th
3
1715405 +/- 14874
55th
3
1714835 +/- 24052
52nd
3
1701625 +/- 9553
52nd
9
1698230 +/- 213078
Median
50th
1676097
40th
3
1597048 +/- 18409
40th
3
1592361 +/- 173176
38th
3
1536164 +/- 15430
29th
3
1491584 +/- 8056
26th
3
1476008 +/- 14009
Low-Tier
25th
< 1473974
25th
4
1468159 +/- 190764
19th
4
1384734 +/- 128792
16th
3
1285631 +/- 29934
13th
3
1220223 +/- 30093
11th
3
921009 +/- 18573
8th
3
330977 +/- 1180
6th
3
328906 +/- 1495
5th
3
322416 +/- 4147
3rd
3
290675 +/- 4479
OpenBenchmarking.orgDistribution Of Public Results - Protocol: Redis - Clients: 50 - Set To Get Ratio: 1:10157 Results Range From 286611 To 3128872 Ops/sec2866113434574003034571495139955708416276876845337413797982258550719119179687631025609108245511393011196147125299313098391366685142353114803771537223159406916509151707761176460718214531878299193514519919912048837210568321625292219375227622123330672389913244675925036052560451261729726741432730989278783528446812901527295837330152193072065312891148121620

Based on OpenBenchmarking.org data, the selected test / test configuration (memtier_benchmark 1.4 - Protocol: Redis - Clients: 50 - Set To Get Ratio: 1:10) has an average run-time of 7 minutes. By default this test profile is set to run at least 3 times but may increase if the standard deviation exceeds pre-defined defaults or other calculations deem additional runs necessary for greater statistical accuracy of the result.

OpenBenchmarking.orgMinutesTime Required To Complete BenchmarkProtocol: Redis - Clients: 50 - Set To Get Ratio: 1:10Run-Time510152025Min: 4 / Avg: 7.29 / Max: 18

Based on public OpenBenchmarking.org results, the selected test / test configuration has an average standard deviation of 0.3%.

OpenBenchmarking.orgPercent, Fewer Is BetterAverage Deviation Between RunsProtocol: Redis - Clients: 50 - Set To Get Ratio: 1:10Deviation246810Min: 0 / Avg: 0.3 / Max: 2

Notable Instruction Set Usage

Notable instruction set extensions supported by this test, based on an automatic analysis by the Phoronix Test Suite / OpenBenchmarking.org analytics engine.

Instruction Set
Support
Instructions Detected
SSE2 (SSE2)
Used by default on supported hardware.
 
CVTSI2SD COMISD UCOMISD MOVAPD DIVSD MULSD CVTTSD2SI ADDSD CVTSS2SD SUBSD MOVDQU MOVDQA MOVD CVTSD2SS SQRTSD
Last automated analysis: 12 March 2022

This test profile binary relies on the shared libraries libevent-2.1.so.7, libcrypto.so.3, libssl.so.3, libm.so.6, libc.so.6.

Tested CPU Architectures

This benchmark has been successfully tested on the below mentioned architectures. The CPU architectures listed is where successful OpenBenchmarking.org result uploads occurred, namely for helping to determine if a given test is compatible with various alternative CPU architectures.

CPU Architecture
Kernel Identifier
Verified On
Intel / AMD x86 64-bit
x86_64
(Many Processors)
ARMv8 64-bit
aarch64
ARMv8 Cortex-A72 4-Core, ARMv8 Neoverse-N1, ARMv8 Neoverse-V1, Apple M1

Recent Test Results

OpenBenchmarking.org Results Compare

4 Systems - 282 Benchmark Results

Intel Core i9-11900K - ASUS ROG MAXIMUS XIII HERO - Intel Tiger Lake-H

Ubuntu 21.10 - 5.19.0-rc2-mmio-stale-data-patched - GNOME Shell 40.5

3 Systems - 172 Benchmark Results

Intel Core i5-12400 - MSI PRO Z690-A WIFI DDR4 - Intel Device 7aa7

Ubuntu 22.04 - 5.15.0-40-generic - GNOME Shell 42.2

3 Systems - 158 Benchmark Results

Intel Core i7-8565U - Dell 0KTW76 - Intel Cannon Point-LP

Ubuntu 22.04 - 5.19.0-rc6-phx-retbleed - GNOME Shell 42.2

3 Systems - 202 Benchmark Results

Intel Core i9-10900K - Gigabyte Z490 AORUS MASTER - Intel Comet Lake PCH

Ubuntu 21.10 - 5.17.0-051700rc7daily20220309-generic - GNOME Shell 40.5

4 Systems - 211 Benchmark Results

AMD Ryzen 9 3900XT 12-Core - MSI MEG X570 GODLIKE - AMD Starship

Ubuntu 22.04 - 5.15.0-40-generic - GNOME Shell 42.2

3 Systems - 311 Benchmark Results

Intel Core i5-12600K - ASUS PRIME Z690-P WIFI D4 - Intel Device 7aa7

Ubuntu 22.04 - 5.19.0-051900rc6daily20220716-generic - GNOME Shell 42.1

3 Systems - 307 Benchmark Results

AMD Ryzen 9 3900X 12-Core - ASUS TUF GAMING X570-PLUS - AMD Starship

Ubuntu 20.04 - 5.11.0-rc1-phx - GNOME Shell 3.36.4

3 Systems - 240 Benchmark Results

Intel Core i7-8700K - ASUS TUF Z370-PLUS GAMING - Intel 8th Gen Core

Ubuntu 22.04 - 5.19.0-rc6-phx-retbleed - GNOME Shell 42.4

1 System - 366 Benchmark Results

2 x AMD EPYC 7713 64-Core - AMD DAYTONA_X - AMD Starship

Ubuntu 22.10 - 6.0.0-060000rc3daily20220904-generic - GNOME Shell

3 Systems - 80 Benchmark Results

AMD Ryzen Threadripper 2950X 16-Core - MSI MEG X399 CREATION - AMD 17h

Ubuntu 22.04 - 5.19.0-051900rc7-generic - GNOME Shell 42.2

3 Systems - 221 Benchmark Results

2 x Intel Xeon Platinum 8380 - Intel M50CYP2SB2U - Intel Device 0998

CentOS Stream 9 - 5.14.0-148.el9.x86_64 - GNOME Shell 40.10

2 Systems - 240 Benchmark Results

2 x Intel Xeon Platinum 8380 - Intel M50CYP2SB2U - Intel Device 0998

CentOS Stream 9 - 5.14.0-148.el9.x86_64 - GNOME Shell 40.10

3 Systems - 81 Benchmark Results

Intel Core i7-1165G7 - Dell 0GG9PT - Intel Tiger Lake-LP

Ubuntu 21.10 - 5.13.0-52-generic - GNOME Shell 40.5

4 Systems - 223 Benchmark Results

AMD Ryzen 7 4700U - LENOVO LNVNB161216 - AMD Renoir

Ubuntu 22.04 - 5.18.8-051808-generic - GNOME Shell 42.2

1 System - 232 Benchmark Results

2 x Intel Xeon Platinum 8380 - Intel M50CYP2SB2U - Intel Device 0998

CentOS Stream 9 - 5.14.0-148.el9.x86_64 - GNOME Shell 40.10

Most Popular Test Results

Find More Test Results