Octane, Speedometer and the Future of Benchmarking in Chrome OS
What the internet is today is a far cry from its humble beginnings as a central hub for a global hyperlinked information system where websites were nothing but simply displayed text and a browser was nothing but a portal to access that.
Now, the internet has expanded exponentially beyond a simple information retrieval center to encompass an entire universe.
This expansion is due to the relentless efforts from companies and development teams whose push is to create and put in place tools to make what was a tedious collection of information both easily shareable and accessible.
One such team is the one behind Google’s V8 engine (a subsidiary within the Chromium project).
V8, a high-performance engine that relies heavily on both javascript and Web assembly (one of the most important building blocks for Chrome as web browser and an OS) was created to negate some of the problems attributed to conventional compiling techniques that included– but not limited to– having a filesystem where the entire program is converted to machine code before being executed.
V8, on the other hand, complies all javascript natively before the code is executed. This method wasn’t only faster than older methods but it was much more reliable as the compiled code is optimized and re-optimized dynamically within the runtime before the execution takes place.
It was the success of V8 that gave birth to Octane (Chrome’s go-to benchmarking suite), a platform that is the cornerstone of this article.
What is Octane?
Octane is a collection of tests within a suite designed under the guidance of the Chromium project that covers a rich list of workloads whose aim is to gauge the performance of javascript frameworks within a web page.
Released in 2012, the Octane Benchmark Suite tests range from a variety of categories (seventeen in total) that include, animation, rendering, DOM transformations, string operations, mathematical calculations, sorting algorithms, graphics performance, as well speed and efficiency of handling memory instructions.
These test (though artificial) were designed to mimic real-world workloads the results of which were later used as the catalyst to improve overall web performance.
Octane which was spliced from V8 but wasn’t the first of its kind by any means as several testing mechanisms came and went before it became mainstream. As suites like Octane or V8 before it, were only able to take off because similar javascript benchmark suits like SunSpider were withering away (or have already disappeared from
Javascript benchmark suites have finite lifespans and are only viable as long as attaining high scores on them does not have diminishing real-world implications. A fate that befell both SunSpider and eventually Octane.
Octane like its predecessors died because most if not all the Javascript implementations required by the benchmark for compiler optimizations that is a precursor to achieving good grades on the tests is being used universally. This lead to scoring high marks on Octane not really being a good indicator of similar performance within real-world scenarios.
As modern libraries and frameworks began incorporating features that are beyond the platform’s capabilities, the incentive to use Octane became far less enticing, the team knew the benchmark suite has lost its edge and in 2017, the V8 team announced through a lengthy blog post, that it was retiring Octane. Though the platform will still be accessible, it would no longer be maintained.
Speedometer
Benchmark suites are like the many-headed serpents in Greek mythology called Hydra because each time one falls, there is always another waiting on the sidelines to take its place.
Despite the V8 team’s insistence that synthetic benchmarks have become less reliable yardsticks to measure real-world Javascript framework implementations, Speedometer seem to have slowly crept into
Speedometer is similar to Octane as it is also Javascript engine benchmarking suite and though it still runs synthetic tests, the platform tries to improve on Octane by being a better approximation of many sites due to its addition of real web frameworks including React, Angular, Ember, and jQuery.
These added tests could be the reason why it was the recommended benchmark suite by the team that created V8 and Octane.
The platform runs tests using TodoMVC which pretends to complete to-do items by adding, completing, and removing them repeatedly using DOM APIs.
This, plus the clever implementation of several other tests helps Speedometer determine how well javascript engines are optimized within a website.
With the plug being pulled on Octane, Speedometer appears to be the next logical alternative for a benchmark suite the closely represents a typical workload within your Chromebook which is why it’s the standard we’ll be using primarily for benchmarks on ItsChromeOS.
Do you think Speedometer is a worthy replacement of Octane? Let us know in the comments!
