Easily take a result set and reproduce the exact execution on a different system.
OpenBenchmarking Performance Classifications (OPC) and other comparison features provide a concise view of any result against the thousands of other results.
When shopping for new computer hardware, users can leverage the vast databases of hardware and software information to find products of interest to them that meets their performance and functional requirements. The OpenBenchmarking.org Product Finder is meant to make the process easy along with the OpenBenchmarking.org Search to ease the process. The OpenBenchmarking.org Performance Classification Indexes are also meant to ease the shopping process by helping users find hardware within a given category that is ranked by overall performance and price.
OpenBenchmarking.org is built around the Phoronix Test Suite, thereby taking advantage of its vast community of testers and years of benchmarking research by Phoronix Media and its partners.
Leveraging OpenBenchmarking.org allows for benchmarks and functional tests from hundreds or thousands of users in a very easy, direct, and real-time manner.
Collect historical results for a software/hardware project or product. By aggregating results on OpenBenchmarking.org you can better determine where regressions or quirks occur as well as spot trends in performance.
OpenBenchmarking.org and the Phoronix Test Suite run on everything from tiny ARM-based plug computers to global cloud-based deployments.
OpenBenchmarking.org and the Phoronix Test Suite support a wide variety of platforms including Linux, Mac OS X, Solaris, *BSD, and Windows.
Free use of OpenBenchmarking.org is encouraged. However, all results are required to be made public or have a short embargo time for which they can be set private. Commercial options are available for organizations wishing to deploy custom or private installations.
Phoronix Media offers a portfolio of extra products and services that provide increased and targeted support for benchmarking and functional testing. These include:
OpenBenchmarking.org is a centralized testing ecosystem. OpenBenchmarking.org provides a collaborative, open test platform with a standardized test profile and suite management system for distributing and standardizing benchmarks.
Users can search for a given software or hardware component and find detailed information on its Linux/BSD/OSX/Solaris performance results, find current pricing information, and compare the performance to like-hardware.
OpenBenchmarking.org not only makes it efficient to gather new tests and suites and to find test results -- whether it be quantitative or qualitative based -- for a given hardware or software configuration, but it also provides a robust feature-set for exploring and discovering new tests and hardware, analyzing results in great detail, communicating over result findings, and to collaborate over future software and hardware tests.
With OpenBenchmarking.org, the user can now easily access updated and/or new test profiles and suites without having to update the test client (assuming the test/suite isn’t dependent upon an XML schema or specification change only found in a later client release).
Access third-party test profiles and suites, that can be created and uploaded by any individual or organization -- there is no "gate keeper" for letting tests in nor are there manual steps any longer to distributing independent tests and suites.
Users can directly compare the performance of multiple test results, find similar hardware components, and discover new hardware that may suit their needs.
Via the OpenBenchmarking.org web interface are a number of analytical features for interpreting multi-way comparisons, finding geometric / aggregate means, calculate performance per price ratios, and examine system log files for given configurations, among other capabilities.
From OpenBenchmarking.org Statistics, view the popularity of difference hardware/software components, different tested kernel/X/distribution configurations for given hardware components, and popular and minimum software versions used in conjunction with selected hardware.
When uploading results to OpenBenchmarking.org, users can optionally attach system logs such as the dmesg, Xorg.0.log, and lspci output. Prominent configuration files such as the xorg.conf can also be uploaded so that others can collaborate and find potential problems or configuration options that may lead to greater performance or other benefits.
Primary focus is obviously on CPU processors, graphics processors, and motherboards, but OpenBenchmarking.org supports many devices, including disk drives, network adapters, and audio adapters. There is also support for analyzing software components such as a particular operating system release, compiler, or kernel version.
OpenBenchmarking.org is completely extensible. Any free software project can take advantage of any of these features for their own testing purposes by simply writing a test profile / suite that compiles with our public specifications. Technically, any benchmarking client can also implement this benchmarking platform and take advantage of this entire framework as well.
Advanced commenting abilities. Users can comment on test results, test profiles, test suites, and even search results. Where this search though differs greatly from forums or Hardware Compatibility Lists is the precise focus and detail. With a forum or HCL-type databases you are dependent upon a user posting a question/comment in a particular section and then finding yourself lucky that a owner of the said hardware will come across that question. With OpenBenchmarking.org, if a potential customer of "Graphics Card XYZ" posts a question asking if "Feature 123" is implemented in the open-source driver for this graphics card, depending upon each user’s notification settings, OpenBenchmarking.org can proactively send a notification to those users that have or had "Graphics Card XYZ" informing them another user has a question, comment, or suggestion based upon their settings. This is leading to the largest global test farm.
OpenBenchmarking.org provides custom user pages to registered account users so they can optionally show off their test systems and results as well as their own custom test profiles and suites.
When the Phoronix Test Suite is installed with .OpenBenchmarking file extension support with MIME recognition, simply clicking on the OpenBenchmarking.org icon from a test result, profile, or suite will automatically launch the Phoronix Test Suite with the given test configuration.
With OpenBenchmarking.org Performance Classifications (OPC), users can see how their own system's performance compares to that of any other system results hosted on OpenBenchmarking.org.
Future work on OpenBenchmarking.org includes analyzing the most common hardware / software configurations, autonomous spotting of performance / compatibility anomalies or quirks, analyzing real-world upgrade trends, auto-parsing and suggestions based upon uploaded configuration files for reaching optimal performance while being self-learning based upon existing uploaded logs and configuration files, embeddable graphs / results onto other web-pages in a rich web format, web-based test profile and suite creation, and an external OpenBenchmarking.org API.
Listed below are several key technologies that OpenBenchmarking.org leverages to provide a rich and extensible user experience.
Phoronix Test Suite: OpenBenchmarking.org leverages pts-core, pts_Graph, nye_Xml, Phodevi, and other PTS-developed components for providing a very integrated experience that shares many of the same features with the test client.
PHXCMS: The Phoronix.com content management system that OpenBenchmarking.org interfaces with for obtaining links to information on Linux drivers, kernel releases, and other information. Community information from the Phoronix Forums can also be accessed. Phoronix.com has been providing Linux information since 2004.
Phoronix LCH: Fundamentals that were learned from this hardware compatibility list that was available to Phoronix users in 2006 for providing Linux hardware compatibility information are used by the OpenBenchmarking.org design.
Phoromatic: This remote test management platform developed for managing large-scale test farms and scheduled test execution queues will offer integration support with OpenBenchmarking.org
External Shopping APIs: Search queries for computer hardware and when performing comparisons will automatically yield relevant hardware pricing data from our shopping partners. This also allows users to search for hardware components in a given price range using the OpenBenchmarking.org Product Finder.
The rest of OpenBenchmarking.org is built upon original intellectual property exclusively for OpenBenchmarking.org. An external API for third-parties to access this massive collection of computer hardware/software performance data, hardware/software information, and other statistics is currently being developed.