How to stop worrying about browser coverage

This post is to introduce browser-coverage, a nifty little CLI tool I have been hacking in Clojure. It looks at your Google Analytics data, you tell it your browser testing plan and it computes how much of your audience it covers in terms of browser support. It’s reached an MVP state but it’s already doing a lot under the hood: data sanitisation, aggregation, employs Logic Programming techniques and more! Have a look at it and let me know what you think. And in case you’re not sure you should care, it has the potential to help us drop support for IE8 for good. Read on for more on that.

This all started from a discussion e-Experience initiated after there were some problems with IE8 users in the Help section. Their concern wasn’t just about this single issue, but also about the overall consistency of our browser support across sky.com. When a user navigates through the various sub-sections of sky.com it would be a really poor experience if one section (e.g. My Sky) works well and then breaks when entering another section (say Help).

After some investigation it became apparent there were some inconsistencies across the various teams, not in a major way, but enough to be considered the cause of the initial problem that started the whole discussion. What made this situation even more difficult was the unreliability of the Omniture browser data presented to us when discussing what should be supported or not. For example in the slide below the two highlighted lines do not make sense and the italic lines lack a precise browser version:

Safari 0.8.2 is not a version that exists, and Chrome Mobile v28 having a 2.5% share (for that version only) was not realistic compared to the 0.5% share it has when looking at Google Analytics data.

While this situation clearly produced some awkward discussions about what it meant to support a browser and what we should officially support, the outcome was clearly positive in terms of establishing some common understanding:

• consistent user experience is paramount and all teams in Sales and Service should share the same testing plan
• a browser is said to be supported if we specifically test it
• this does not mean we should test each version possible: most browsers are upward compatible meaning testing a version covers for subsequent versions
• it’s okay to only support functionally but not cosmetically older versions of certain browsers¹,
• rather than covering for all browsers with a share above 0.1% of our audience, we should aim to cover at least 95% of our audience
• at some point it would be nice to have a consistent way to message the user about an unsupported or partially supported browser
• Omniture data is not reliable enough notably in terms of browser versions

What this means is that we need to devise a common test plan that covers 95% of our audience but without using Omniture data. Gladly Google Analytics data came to the rescue: the Contact Us section of sky.com has Google Analytics enabled and I was quickly able to produce a better picture of our audience in terms of browsers. Although not completely perfect, Google Analytics data is much more reliable in reporting precise version numbers, which meant I was able to produce a test plan which almost had the required 95% coverage. The major difference with what is currently tested is that we needed to specifically test for IE8 and also test some more mobile browsers.

The trouble however with that manual approach was twofold:

• some data needed to be sanitised and aggregated according to upward compatibility rules, doing so would show the expected 95%
• it’s a one-off calculation, new browsers and versions constantly appear and we need to constantly monitor our coverage

I couldn’t see myself doing this manual data shuffling on a regular basis, so it had to be automated as much as possible. Which is why I developped the browser-coverage tool for which this post is a rather long introduction. So to cut the story short, I developed a command line tool that takes as input a spreadsheet containing the testing plan and various configuration data (sanitisation, coverage rules, etc.) and computes the coverage using Google Analytics data.

The README contains further details that would be pointless to reproduce here. What is interesting to mention however is the power to be able to show interactively how changing some lines in a test plan captured in a spreadsheet impacts our coverage: it enabled a more productive discussion with our main stakeholder Richard Cartland. More precisely, during an informal demo of this tool Richard agreed to drop our support for IE8 if we still manage to cover for 96% of our audience. Why 96% and not 95% as mentioned before? Because that’s what we can achieve with the folowwing test plan based on Contact Us March data:

So we now have a case for getting rid of IE8 support, at least on the service side because shop has stronger coverage requirements. For that we would need to test more older mobile browsers such as Android Browser 1.5 and Safari on iOS 5.

I will be presenting this at the next Scrum master meeting to get feedback, and will report on this blog post.

– Francois Rey

[Edit, 21 May 2015] At the Scrum Master meeting it was agreed we would give each team 2 weeks to assess how they can commit to this testing plan. More on this early May.