Numenta Anomaly Benchmark

Numenta Anomaly Benchmark (NAB) is a benchmark for evaluating algorithms for anomaly detection in streaming, real-time applications. It is comprised of over 50 labeled real-world and artificial timeseries data files plus a novel scoring mechanism designed for real-time applications. This test profile currently measures the time to run various detectors.

Learn more about this test at the upstream project site: https://github.com/numenta/NAB.

This utility test profile is maintained by Michael Larabel. This OpenBenchmarking.org test profile was created on 9 November 2018 and last updated on 28 February 2020. To run this test with the Phoronix Test Suite, the basic command is: phoronix-test-suite benchmark numenta-nab.

This test has an average install time of 1 minute, 47 seconds and an average run-time of 18 minutes, 30 seconds.

10k+ Downloads

Installation StatsTest Completion StatsPublic Result UploadsOpenBenchmarking.orgEventsNumenta Anomaly Benchmark Popularitypts/numenta-nab2018.112018.122019.012019.022019.032019.042019.052019.062019.072019.082019.092019.102019.112019.122020.012020.022020.032020.042020.052020.062020.072020.082020.092020.102020.11120240360480600

* Data based on those opting to upload their test results to OpenBenchmarking.org and users enabling the opt-in anonymous statistics reporting.

Windowed Gaussian21.9%EXPoSE13.8%Relative Entropy21.8%Earthgecko Skyline20.4%Bayesian Changepoint22.1%Detector Option PopularityOpenBenchmarking.org

Revision History

pts/numenta-nab-1.1.0   [View Source]   28 Feb 2020 12:04 EST
Update against NAB 1.1 upstream for Python 3 compatibility.

pts/numenta-nab-1.0.0   [View Source]   09 Nov 2018 11:55 EST
Initial commit of Numenta Anomaly Benchmark

Suites Using This Test

Machine Learning

CPU Massive

Server CPU Tests

Python

Performance Metrics

Analyze Test Configuration:

Numenta Anomaly Benchmark 1.1

Detector: Bayesian Changepoint

OpenBenchmarking.org metrics for this test profile configuration based on 407 public samples since 28 February 2020 with the latest data as of 1 December 2020.

Below is an overview of the generalized performance for components where there is sufficient statistically significant data based upon user-uploaded results. It is important to keep in mind particularly in the Linux/open-source space there can be vastly different OS configurations, with this overview intended to offer just general guidance as to the performance expectations.

OpenBenchmarking.orgDistribution Of Public Results - Detector: Bayesian Changepoint407 Results Range From 21 To 1732 Seconds2162103144185226267308349390431472513554595636677718759800841882923964100510461087112811691210125112921333137414151456149715381579162016611702174370140210280350

Based on OpenBenchmarking.org data, the selected test / test configuration (Numenta Anomaly Benchmark 1.1 - Detector: Bayesian Changepoint) has an average run-time of 4 minutes. By default this test profile is set to run at least 3 times but may increase if the standard deviation exceeds pre-defined defaults or other calculations deem additional runs necessary for greater statistical accuracy of the result.

OpenBenchmarking.orgMinutesTime Required To Complete BenchmarkDetector: Bayesian ChangepointRun-Time714212835Min: 1 / Avg: 3.12 / Max: 32

Based on public OpenBenchmarking.org results, the selected test / test configuration has an average standard deviation of 0.6%.

OpenBenchmarking.orgPercent, Fewer Is BetterAverage Deviation Between RunsDetector: Bayesian ChangepointDeviation246810Min: 0 / Avg: 0.64 / Max: 3

Recent Test Results

Compare

1 System - 41 Benchmark Results

Intel Core i7-6700 - ASUS Z170I PRO GAMING - Intel Xeon E3-1200 v5

Fedora 33 - 5.8.16-300.fc33.x86_64 - Cinnamon 4.6.7

1 System - 92 Benchmark Results

AMD Ryzen 9 5900X 12-Core - ASRock X570 Taichi - AMD Starship

Ubuntu 20.04 - 5.4.0-53-generic - GNOME Shell 3.36.4

1 System - 22 Benchmark Results

ARMv7 Cortex-A72 - BCM2711 Raspberry Pi 4 Model B Rev 1.2 - 2048MB

Raspbian 10 - 5.4.79-s3694257+ - LXDE

1 System - 17 Benchmark Results

ARMv7 rev 3 - BCM2711 - 855MB

Raspbian 10 - 5.4.75-v7l-s3791004+ - LXDE 0.10.0

1 System - 17 Benchmark Results

ARMv7 rev 3 - BCM2711 - 8192MB

Raspbian 10 - 5.4.77-s3634235+ - LXDE 0.10.0

1 System - 21 Benchmark Results

ARMv7 Cortex-A72 - BCM2711 Raspberry Pi 4 Model B Rev 1.2 - 4096MB

Raspbian 10 - 5.4.77-v7l-S3612985+ - LXDE

1 System - 80 Benchmark Results

AMD EPYC 7401P 24-Core - Supermicro H11SSW-NT - AMD 17h

Ubuntu 20.04 - 5.4.0-53-generic - GCC 9.3.0

1 System - 21 Benchmark Results

ARMv7 Cortex-A72 - BCM2711 Raspberry Pi 4 Model B Rev 1.1 - 4096MB

Raspbian 10 - 5.4.75-v7l-s3569505+ - LXDE 0.10.0

1 System - 21 Benchmark Results

ARMv7 Cortex-A72 - BCM2711 Raspberry Pi 4 Model B Rev 1.4 - 8GB

Raspbian 10 - 5.4.75-s3597958+ - LXDE 0.10.0

1 System - 17 Benchmark Results

ARMv7 rev 3 - BCM2711 - 4096MB

Raspbian 10 - 5.4.74-v7ls3775590-g2ae9829c3da5 - LXDE 0.10.0

1 System - 17 Benchmark Results

ARMv7 rev 3 - BCM2711 - 2048MB

Raspbian 10 - 5.4.77-v7l-gb8681a08ba16 - X Server plymouth.ignore-serial-consoles

2 Systems - 17 Benchmark Results

ARMv7 rev 3 - BCM2711 - 4096MB

Raspbian 10 - 5.4.72-v7l+ - LXDE 0.10.0

Most Popular Test Results

Compare

12 Systems - 593 Benchmark Results

AMD Ryzen 9 3950X 16-Core - ASUS ROG CROSSHAIR VIII HERO - AMD Starship

Ubuntu 20.04 - 5.8.0-050800daily20200622-generic - GNOME Shell 3.36.2

11 Systems - 217 Benchmark Results

AMD Ryzen 9 5900X 12-Core - ASUS ROG CROSSHAIR VIII HERO - AMD Starship

Ubuntu 20.04 - 5.9.0-050900-generic - GNOME Shell 3.36.4

2 Systems - 475 Benchmark Results

AMD Ryzen Threadripper 3970X 32-Core - ASUS ROG ZENITH II EXTREME - AMD Starship

Ubuntu 19.10 - 5.3.0-18-generic - GNOME Shell 3.34.1

6 Systems - 733 Benchmark Results

Intel Core i5-10600K - ASUS PRIME Z490M-PLUS - Intel Comet Lake PCH

Ubuntu 20.04 - 5.8.0-050800daily20200622-generic - GNOME Shell 3.36.2

3 Systems - 301 Benchmark Results

Intel Core i5-7600K - Gigabyte Z270M-D3H-CF - Intel Xeon E3-1200 v6

Ubuntu 20.04 - 5.4.0-40-generic - GNOME Shell 3.36.3

12 Systems - 229 Benchmark Results

AMD Ryzen 7 3800XT 8-Core - ASUS ROG CROSSHAIR VIII HERO - AMD Starship

Ubuntu 20.04 - 5.9.0-050900-generic - GNOME Shell 3.36.4

3 Systems - 777 Benchmark Results

Intel Core i3-10100 - ASUS PRIME Z490M-PLUS - Intel Comet Lake PCH

Ubuntu 20.04 - 5.7.0-rc6-amd-energy - GNOME Shell 3.36.2

3 Systems - 171 Benchmark Results

Intel Xeon E3-1270 v5 - ASUS E3 PRO GAMING V5 - Intel Xeon E3-1200 v5

Clear Linux OS 31470 - 5.3.8-854.native - GNOME Shell 3.34.1

3 Systems - 208 Benchmark Results

Intel Xeon E3-1260L v5 - ASRock E3V5 WS - Intel Xeon E3-1200 v5

Ubuntu 20.10 - 5.8.0-20-generic - GNOME Shell 3.38.0

5 Systems - 385 Benchmark Results

AMD Ryzen 9 3950X 16-Core - ASUS ROG CROSSHAIR VIII HERO - AMD Starship

Ubuntu 20.04 - 5.7.0-999-generic - GNOME Shell 3.36.1

Find More Test Results


User Comments

Post A Comment