Flaky tests cost more than your device farm

What should a mobile app testing strategy actually optimise for?

A mobile app testing strategy should optimise two costs: the cost of a defect your users find, and the cost of a failure your CI invents. Framework choice, coverage percentages and the size of the device farm are all downstream of those two numbers. Most teams over-buy device breadth and under-fund test stability, then watch release confidence fall while the QA bill rises.

Buying automation is no longer the hard part. Automated testing already held 46.05 percent of the mobile app testing services market in 2025,¹ and that market is forecast to grow from $9.02 billion in 2026 to $19.84 billion by 2031.² The hard part is making the automated signal trustworthy enough that a red build actually stops a release.

Why does the mobile automation pyramid look different from web?

The test automation pyramid puts a large base of fast unit tests under a smaller band of integration tests and a thin layer of slow end-to-end UI tests. On web that shape holds up. On mobile it needs rebalancing, because the failures that reach users are gestures, OS permission dialogs, background and resume behaviour, storage migrations and network transitions. None of those live in a unit test.

The failure data supports the rebalance. Mobile test suites fail 20 to 30 percentage points more often than equivalent web suites.³ A pyramid copied across from web application work therefore under-invests in precisely the layer where mobile breaks.

Those shares are a starting allocation, not a published benchmark; no public dataset fixes a correct ratio. Shift weight toward the base when the app is logic-heavy, meaning pricing, sync or offline state. Shift it toward the device layer when the app is platform-heavy, meaning camera, background location, biometrics or deep links.

How many devices does an iOS and Android test plan really need?

Far fewer than the fragmentation figures imply. Over 24,000 distinct Android device variants existed as of 2024, with Samsung accounting for roughly 40 percent of them.⁹ That number gets quoted to justify enormous device matrices. Real traffic points the other way: over 68 percent of global Android sessions occur on fewer than 20 phone models.¹⁰

Build the matrix from your own analytics, in three tiers. Tier one is the models and OS versions covering the bulk of your sessions, tested on real hardware every release. Tier two is the next band, run nightly on a cloud device farm. Tier three is everything else, which gets crash monitoring and no dedicated suite. Coverage past that threshold buys reassurance, not risk reduction.

The two platforms also need different budgets. Android held 68 percent of global mobile OS market share against 31.94 percent for iOS in May 2026,⁷ and carries most of the hardware spread. Its risk is device and OS-version variety. On iOS the device list is short, but the risk sits in permission prompts, platform-version behaviour changes and privacy rules that alter what your code is allowed to do between releases. One shared test plan for native iOS and Android builds reliably under-serves one of them, usually iOS.

What do flaky tests cost, and how should you price them?

A flaky test passes and fails on unchanged code, normally because of timing, network or environment nondeterminism. Mobile makes all three worse: emulator boot variance, animation timing, real network conditions and permission state carried between runs. The share of teams reporting flakiness rose from 10 percent in 2022 to 26 percent by mid-2025.⁴

The expensive part is not the rerun, it is the triage. An engineer investigating a red build has to prove the failure is fake before dismissing it, and that proof costs the same whether the bug is real or not. Across a sample of more than 1,000 teams, an average 100-person engineering team loses about $2.6 million a year to flaky test failures.⁴ A separate analysis puts flaky failures at over 8 percent of total development time in enterprise teams, around $120,000 a year for a 50-engineer team.⁵ The two figures use different team sizes and methods, so read the direction rather than the gap between them.

Track cost per flake, not flake rate. The rate tells you how untidy the suite is. The cost tells you whether to fix it this sprint.

The operating policy is short. Quarantine a test on its second unexplained failure and assign a named owner. Give the owner ten working days to fix or delete it. Record median triage minutes per flake and publish that number next to build duration, because a suite nobody trusts is worse than a smaller suite everybody does.

Where does crash stability belong in the release gate?

In every release, not only at launch. 88 percent of users say they will abandon an app after encountering bugs and glitches,⁶ and apps that crash carry a 30 to 40 percent higher uninstall rate than stable ones.⁷

The shape of that churn matters more than the headline. A single reported crash is associated with a 15.4 percent 30-day uninstall rate, while users who hit two or three crashes reach 50.4 percent.⁷ For scale, the all-app Android 30-day uninstall benchmark was 46.1 percent in 2024, down from 46.9 percent in 2023.⁸

Read that chart carefully. The first two bars are crash-conditioned uninstall rates from a user survey; the third is a market-wide benchmark across all apps. They share an outcome but not a denominator, so the comparison is directional. The usable finding is the slope: churn is not linear in crashes, and the jump between the first crash and the third is where retention is lost. That argues for gating each staged rollout on crash-free session rate, with an automatic halt threshold, rather than treating stability as a pre-launch QA milestone.

How do you stop CI bottlenecking on device availability?

Treat device time as the scarcest resource in the pipeline and spend it last. Four gates handle almost every mobile repository:

1. Commit. Lint, static analysis and unit tests. No device, no emulator, under five minutes.
2. Pull request. Component and integration tests on emulators and simulators, sharded across parallel workers.
3. Merge queue. Device-level end to end on tier one hardware, smoke scope only, capped by wall-clock budget.
4. Nightly and pre-release. Full device matrix on the cloud farm, plus performance, security and accessibility passes.

Real device against emulator against cloud farm is not a single choice. Emulators buy speed and volume, so they carry the integration band. Real devices are for anything touching sensors, biometrics, camera, background execution, battery or real radios, because those are the behaviours emulators approximate rather than reproduce. Cloud farms buy breadth on demand, which is why cloud testing was a $12.83 billion market in 2025.¹¹ Owning a small rack still pays for tier one, where debugging speed matters and the model list barely changes from quarter to quarter.

How should the budget split across manual, automated and specialist testing?

The three answer different questions, so splitting them by percentage is the wrong frame. Automation owns regression, meaning everything already decided and expected to keep working. Manual and exploratory testing owns first-run experience, new features and any case where you do not yet know what the assertion should be. Crowd testing owns what you cannot rent: real carriers, payment methods, locales and long-tail hardware in the markets you sell into.

Security and accessibility sit badly as a budget percentage because both are pass or fail gates rather than coverage targets. Make them fixed checklist items instead. For security that means keystore and keychain handling, certificate pinning, permission scope and data at rest and in transit. For accessibility it means dynamic type, contrast, screen reader labels, focus order and touch target size. Anything expressed as a percentage of sprint capacity will lose to a feature.

Where should a mobile app testing strategy start?

With measurement, not tooling. Three moves fit inside a fortnight. Pull the device model and OS version distribution from your own analytics and cut the matrix to three tiers. Instrument the suite for flakiness by recording every rerun, tagging every unexplained failure and starting a cost-per-flake figure from median triage minutes. Move the device-level suite off the pull request path onto the merge queue, and gate the staged rollout on crash-free sessions.

The pyramid rebalance takes a quarter, because it means writing tests you do not have rather than deleting tests you do. That work sits inside how the rest of delivery is sequenced, which is why testing decisions are best made alongside the wider software development life cycle and the rest of the engineering delivery model rather than bolted on at the end of it.

Written by the group's editorial team with the practice leads who run these builds. Reviewed before publish. Spotted an error? Tell us and we will fix it.