The Full Feature Set

OpenBenchmarking.org is a centralized testing ecosystem. OpenBenchmarking.org provides a collaborative, open test platform with a standardized test profile and suite management system for distributing and standardizing benchmarks.

Users can search for a given software or hardware component and find detailed information on its Linux/BSD/OSX/Solaris performance results, find current pricing information, and compare the performance to like-hardware.

OpenBenchmarking.org not only makes it efficient to gather new tests and suites and to find test results -- whether it be quantitative or qualitative based -- for a given hardware or software configuration, but it also provides a robust feature-set for exploring and discovering new tests and hardware, analyzing results in great detail, communicating over result findings, and to collaborate over future software and hardware tests.

With OpenBenchmarking.org, the user can now easily access updated and/or new test profiles and suites without having to update the test client (assuming the test/suite isn’t dependent upon an XML schema or specification change only found in a later client release).

Access third-party test profiles and suites, that can be created and uploaded by any individual or organization -- there is no "gate keeper" for letting tests in nor are there manual steps any longer to distributing independent tests and suites.

Users can directly compare the performance of multiple test results, find similar hardware components, and discover new hardware that may suit their needs.

Via the OpenBenchmarking.org web interface are a number of analytical features for interpreting multi-way comparisons, finding geometric / aggregate means, calculate performance per price ratios, and examine system log files for given configurations, among other capabilities.

From OpenBenchmarking.org Statistics, view the popularity of difference hardware/software components, different tested kernel/X/distribution configurations for given hardware components, and popular and minimum software versions used in conjunction with selected hardware.

When uploading results to OpenBenchmarking.org, users can optionally attach system logs such as the dmesg, Xorg.0.log, and lspci output. Prominent configuration files such as the xorg.conf can also be uploaded so that others can collaborate and find potential problems or configuration options that may lead to greater performance or other benefits.

Primary focus is obviously on CPU processors, graphics processors, and motherboards, but OpenBenchmarking.org supports many devices, including disk drives, network adapters, and audio adapters. There is also support for analyzing software components such as a particular operating system release, compiler, or kernel version.

OpenBenchmarking.org is completely extensible. Any free software project can take advantage of any of these features for their own testing purposes by simply writing a test profile / suite that compiles with our public specifications. Technically, any benchmarking client can also implement this benchmarking platform and take advantage of this entire framework as well.

Advanced commenting abilities. Users can comment on test results, test profiles, test suites, and even search results. Where this search though differs greatly from forums or Hardware Compatibility Lists is the precise focus and detail. With a forum or HCL-type databases you are dependent upon a user posting a question/comment in a particular section and then finding yourself lucky that a owner of the said hardware will come across that question. With OpenBenchmarking.org, if a potential customer of "Graphics Card XYZ" posts a question asking if "Feature 123" is implemented in the open-source driver for this graphics card, depending upon each user’s notification settings, OpenBenchmarking.org can proactively send a notification to those users that have or had "Graphics Card XYZ" informing them another user has a question, comment, or suggestion based upon their settings. This is leading to the largest global test farm.

OpenBenchmarking.org provides custom user pages to registered account users so they can optionally show off their test systems and results as well as their own custom test profiles and suites.

When the Phoronix Test Suite is installed with .OpenBenchmarking file extension support with MIME recognition, simply clicking on the OpenBenchmarking.org icon from a test result, profile, or suite will automatically launch the Phoronix Test Suite with the given test configuration.

With OpenBenchmarking.org Performance Classifications (OPC), users can see how their own system's performance compares to that of any other system results hosted on OpenBenchmarking.org.

Future work on OpenBenchmarking.org includes analyzing the most common hardware / software configurations, autonomous spotting of performance / compatibility anomalies or quirks, analyzing real-world upgrade trends, auto-parsing and suggestions based upon uploaded configuration files for reaching optimal performance while being self-learning based upon existing uploaded logs and configuration files, embeddable graphs / results onto other web-pages in a rich web format, web-based test profile and suite creation, and an external OpenBenchmarking.org API.