Testing Tools discussed: Test Coverage Index by Perfecto Mobile
12/01/2015
By Greg Jotham, Chief Quality Auditor at AQuA
There’s a tendency to think of testing tools as physical things – hardware or software, essentially. But sometimes information that allows you to plan your testing strategy efficiently is just as valuable. That’s very much the case for the item I’m highlighting here – the Mobile Test Coverage Index issued by Perfecto Mobile.
Cost vs. coverage
AQuA Endorsed Member, Perfecto Mobile are well aware of the conflict between testing coverage and testing cost – developers need to make sure their app works on the widest range of relevant devices, but without inflating the cost of testing to the point where it could impact the viability of the product.
Whether you’re maintaining a stable of testing devices or hiring time on remote testing facilities, either approach can have significant overheads when trying to achieve wider coverage. And if you’re only testing on the devices you can rustle up amongst co-workers, friends and family, you’re inevitably going to get caught out by a quirk of some popular (and probably expensive) device that none of your immediate circle owns. So wider coverage is good, but costs are hard to contain.
Intelligence
That’s why intelligence – in the sense of information – about what devices and OS versions are currently most popular in your target market, can be a key factor in using your testing budget wisely. Intelligently, if you like.
Perfecto Mobile have developed a methodology for determining the best range of devices to be used for testing that will reach a particular level of coverage. To ease cost vs. coverage decisions, the recommendations are broken down by groupings that can provide coverage of the top 30%, 50% or 80% of a given market. Data is taken from existing market research and also Perfecto’s own data sources that cover over 4,000 device profiles and more than 360,000 user profiles.
The Index covers multiple mobile Operating Systems – Android, iOS, Windows Phone and BlackBerry – as well as screen sizes and resolutions, different hardware capabilities, and the best mix of devices to target for coverage in the US and the five largest EU countries, with breakouts to give separate detail for Germany and the UK. Coverage breakdown by OS versions is also given, as well as “new and emerging” devices and OSs, to aid forward planning.
Coverage recommendations
The outcome is a clear recommendation of which devices to target for best coverage in a particular market. With a list of between 10 and 16 devices (depending on OS and region), it’s possible to cover the top 30% of the market. Increasing the stable to between 17 and 25 covers the top 50%, and a collection of 26 to 32 devices will cover the top 80%.
Everything is laid out in a straightforward and easy to read tabular format, and the document is reissued at quarterly intervals to keep it current. Best of all, it’s free of charge.
Developers should make use of this information
Choosing the right devices to test against is vital for confidence in the quality of the customer experience, and the Mobile Test Coverage Index is an invaluable short-cut to identifying what those devices should be. Any developer targeting mobile users in the US and Europe should make use of it to ensure their testing strategy is correctly targeted, at manageable cost.
Take a look at Perfecto Mobile's Mobile Test Coverage index
› back
