Numenta Anomaly Benchmark

Numenta Anomaly Benchmark (NAB) is a benchmark for evaluating algorithms for anomaly detection in streaming, real-time applications. It is comprised of over 50 labeled real-world and artificial timeseries data files plus a novel scoring mechanism designed for real-time applications. This test profile currently measures the time to run various detectors.

To run this test with the Phoronix Test Suite, the basic command is: phoronix-test-suite benchmark numenta-nab.

Project Site

github.com

Test Created

9 November 2018

Last Updated

28 February 2020

Test Maintainer

Michael Larabel 

Test Type

System

Average Install Time

4 Minutes, 12 Seconds

Average Run Time

18 Minutes, 30 Seconds

Test Dependencies

Python

Accolades

20k+ Downloads

Supported Platforms


Public Result UploadsReported Installs*Test Completions*OpenBenchmarking.orgEventsNumenta Anomaly Benchmark Popularity Statisticspts/numenta-nab2018.112018.122019.012019.022019.032019.042019.052019.062019.072019.082019.092019.102019.112019.122020.012020.022020.032020.042020.052020.062020.072020.082020.092020.102020.112020.122021.012021.022021.032021.042021.055001000150020002500
* Data based on those opting to upload their test results to OpenBenchmarking.org and users enabling the opt-in anonymous statistics reporting while running benchmarks from an Internet-connected platform.
Data current as of Sat, 08 May 2021 14:42:07 GMT.
Windowed Gaussian19.5%EXPoSE12.1%Relative Entropy23.2%Earthgecko Skyline21.9%Bayesian Changepoint23.3%Detector Option PopularityOpenBenchmarking.org

Revision History

pts/numenta-nab-1.1.0   [View Source]   Fri, 28 Feb 2020 12:04:17 GMT
Update against NAB 1.1 upstream for Python 3 compatibility.

pts/numenta-nab-1.0.0   [View Source]   Fri, 09 Nov 2018 11:55:56 GMT
Initial commit of Numenta Anomaly Benchmark

Suites Using This Test

Machine Learning

CPU Massive

Server CPU Tests

HPC - High Performance Computing

Python


Performance Metrics

Analyze Test Configuration:

Numenta Anomaly Benchmark 1.1

Detector: EXPoSE

OpenBenchmarking.org metrics for this test profile configuration based on 309 public results since 28 February 2020 with the latest data as of 2 May 2021.

Below is an overview of the generalized performance for components where there is sufficient statistically significant data based upon user-uploaded results. It is important to keep in mind particularly in the Linux/open-source space there can be vastly different OS configurations, with this overview intended to offer just general guidance as to the performance expectations.

Component
Percentile Rank
# Matching Public Results
Seconds (Average)
Mid-Tier
75th
> 586
70th
11
627 +/- 22
63rd
4
671 +/- 3
61st
4
681 +/- 14
59th
5
695 +/- 13
51st
6
749 +/- 6
Median
50th
755
48th
5
787 +/- 36
42nd
3
838 +/- 2
36th
4
940 +/- 123
35th
3
943 +/- 13
34th
5
963 +/- 9
30th
3
1013 +/- 9
Low-Tier
25th
> 1091
12th
6
1429 +/- 15
10th
5
1452 +/- 124
OpenBenchmarking.orgDistribution Of Public Results - Detector: EXPoSE307 Results Range From 28 To 15995 Seconds2834866898813081628194822682588290832283548386841884508482851485468578861086428674870687388770880288348866889889308962899481026810588109081122811548118681218812508128281314813468137881410814428147481506815388157081602820406080100

Based on OpenBenchmarking.org data, the selected test / test configuration (Numenta Anomaly Benchmark 1.1 - Detector: EXPoSE) has an average run-time of 49 minutes. By default this test profile is set to run at least 3 times but may increase if the standard deviation exceeds pre-defined defaults or other calculations deem additional runs necessary for greater statistical accuracy of the result.

OpenBenchmarking.orgMinutesTime Required To Complete BenchmarkDetector: EXPoSERun-Time4080120160200Min: 1 / Avg: 48.63 / Max: 226

Based on public OpenBenchmarking.org results, the selected test / test configuration has an average standard deviation of 1%.

OpenBenchmarking.orgPercent, Fewer Is BetterAverage Deviation Between RunsDetector: EXPoSEDeviation3691215Min: 0 / Avg: 1.05 / Max: 8

Recent Test Results

OpenBenchmarking.org Results Compare

1 System - 26 Benchmark Results

AMD Ryzen 9 5950X 16-Core - Gigabyte B550M AORUS PRO - AMD Starship

Ubuntu 20.04 - 5.11.13-051113-generic - GNOME Shell 3.36.7

1 System - 272 Benchmark Results

1 System - 272 Benchmark Results

1 System - 92 Benchmark Results

AMD Ryzen 5 5600X 6-Core - ASRock X570 Phantom Gaming-ITX/TB3 - AMD Device 1480

Ubuntu 18.04 - 5.4.0-70-generic - GNOME Shell 3.28.4

1 System - 111 Benchmark Results

2 x Intel Xeon E5-2678 v3 - HUANANZHI X99-T8D - Intel Xeon E7 v3

Debian 10 - 4.19.0-13-amd64 - GCC 8.3.0

1 System - 241 Benchmark Results

AMD Ryzen 7 3700X 8-Core - ASRock X570 Phantom Gaming-ITX/TB3 - AMD Starship

Ubuntu 20.10 - 5.8.0-41-lowlatency - X Server 1.20.9

1 System - 23 Benchmark Results

AMD Ryzen 5 4600H - HUAWEI HLYL-WXX9-PCB - AMD Renoir Root Complex

Ubuntu 20.10 - 5.8.0-25-generic - Budgie

2 Systems - 6 Benchmark Results

AMD Ryzen 9 5900X 12-Core - ASUS ROG CROSSHAIR VIII HERO - AMD Starship

Ubuntu 20.10 - 5.11.0-051100rc2daily20210105-generic - GNOME Shell 3.38.1

1 System - 6 Benchmark Results

AMD Ryzen 9 3900XT 12-Core - MSI MEG X570 GODLIKE - AMD Starship

Ubuntu 20.10 - 5.11.0-rc1-phx - GNOME Shell 3.38.1

4 Systems - 210 Benchmark Results

POWER9 - PowerNV T2P9D01 REV 1.01 - 64GB

Ubuntu 20.10 - 5.9.10-050910-generic - X Server

2 Systems - 85 Benchmark Results

Intel Core i7-10510U - LENOVO 20U9CTO1WW - Intel Comet Lake PCH-LP

Fedora 33 - 5.9.16-200.fc33.x86_64 - KDE Plasma 5.20.4

1 System - 83 Benchmark Results

Intel Core i5-1135G7 - Dell 0THX8P - Intel Device a0ef

Ubuntu 20.04 - 5.6.0-1036-oem - GNOME Shell 3.36.4

Most Popular Test Results

OpenBenchmarking.org Results Compare

2 Systems - 1708 Benchmark Results

Intel Core i3-10100 - ASUS PRIME Z490M-PLUS - Intel Comet Lake PCH

Ubuntu 20.04 - 5.7.0-rc6-amd-energy - GNOME Shell 3.36.2

6 Systems - 733 Benchmark Results

AMD Ryzen 9 3950X 16-Core - ASUS ROG CROSSHAIR VIII HERO - AMD Starship

Ubuntu 20.04 - 5.8.0-050800daily20200622-generic - GNOME Shell 3.36.2

3 Systems - 777 Benchmark Results

Intel Core i3-10100 - ASUS PRIME Z490M-PLUS - Intel Comet Lake PCH

Ubuntu 20.04 - 5.7.0-rc6-amd-energy - GNOME Shell 3.36.2

3 Systems - 208 Benchmark Results

Intel Xeon E3-1260L v5 - ASRock E3V5 WS - Intel Xeon E3-1200 v5

Ubuntu 20.10 - 5.8.0-20-generic - GNOME Shell 3.38.0

5 Systems - 385 Benchmark Results

2 x Intel Xeon E5-2670 0 - Supermicro X9DRD-iF - Intel Xeon E5

Debian 10 - 4.19.0-9-amd64 - GCC 8.3.0

2 Systems - 85 Benchmark Results

Intel Core i7-10510U - LENOVO 20U9CTO1WW - Intel Comet Lake PCH-LP

Fedora 33 - 5.9.16-200.fc33.x86_64 - KDE Plasma 5.20.4

3 Systems - 259 Benchmark Results

AMD EPYC 7F72 24-Core - ASRockRack EPYCD8 - AMD Starship

Ubuntu 20.04 - 5.9.0-rc4-pts - GNOME Shell 3.36.4

1 System - 809 Benchmark Results

AMD Ryzen 9 3900X 12-Core - ASUS TUF GAMING X570-PLUS - AMD Starship

Ubuntu 20.04 - 5.4.0-14-generic - GNOME Shell 3.35.91

1 System - 908 Benchmark Results

AMD Ryzen Threadripper 3990X 64-Core - System76 Thelio Major - AMD Starship

Ubuntu 20.04 - 5.6.0-999-generic - GNOME Shell 3.36.0

1 System - 83 Benchmark Results

Intel Core i5-1135G7 - Dell 0THX8P - Intel Device a0ef

Ubuntu 20.04 - 5.6.0-1036-oem - GNOME Shell 3.36.4

Find More Test Results