Certifying a banking app across a fragmented Android fleet with 15 test days
The mobile team at a retail bank must certify a new release of its Android and iOS app before a regulator-set deadline in six weeks, and only 15 test days are available to one tester. Analytics from the previous release show Android 13 at 41% of sessions, Android 12 at 22%, Android 11 at 12%, iOS 17 at 9%, iOS 16 at 8%, iOS 15 at 5%, with the remainder spread over older versions. Twenty-eight distinct Android OEM/OS combinations appear in the dashboard, although the top five cover 63% of sessions. Two OEMs - a mid-range Chinese brand and a tablet-only line - generate 70% of crash reports, mostly around biometric login and camera-based cheque deposit. The team owns six physical Android devices, four iPhones and a cloud device farm subscription limited to 300 minutes per month. The product owner wants every combination in the dashboard tested, including the tablet line at 1.4% of sessions.
1. Produce a risk-based device selection for the 15 test days. State which devices and OS versions you keep, what you drop, and how you split work between physical devices and the cloud farm. Justify each decision.
Read the worked answer
With 15 days and one tester, selection must follow real usage coverage first, risk second, hardware dependence third - not the length of the analytics list.
Coverage groups (target ≥90% of sessions):
- Android 13 (41%) + Android 12 (22%) = 63%, two devices.
- Android 11 (12%) - one device, ideally the crash-heavy OEM.
- iOS 17 (9%) + iOS 16 (8%) = 17%, two iPhones. That is five OS-version slots. Add one older Android (e.g. Android 10) and iOS 15 for smoke only, and document the ~4% tail as accepted residual risk.
Risk-driven additions: both crash-concentrated OEMs get full functional runs. Crash concentration is a better predictor of field defects than session share - 70% of crashes from two OEMs is a signal you ignore at your peril. Add one small low-RAM Android and one large-screen device, chosen from the OEM list, to cover screen/density and memory classes.
Physical vs cloud:
- Physical devices for biometric login (enrolment, lockout, changed fingerprint), camera/cheque capture (autofocus, glare, macro), performance, battery, sensors, and anything needing a real SIM or incoming call.
- Cloud farm for breadth smoke on the long tail - one scripted 8–10 minute script per combination. 300 minutes buys roughly 30 smoke runs; budget them and never run long exploratory sessions there.
Write the matrix and the exclusions into the test strategy with sign-off. Auditors and regulators ask why a combination was skipped, not only what was tested.
2. The product owner insists that all 28 Android OEM/OS combinations must be tested before the release. Draft your response, including the arithmetic you would use to show the trade-off, and what you would offer instead.
Read the worked answer
Put the arithmetic on one page, in the units the product owner cares about.
- 28 combinations. Even a 20-minute smoke each = 9.3 device-hours, plus install, data reset, setup and triage - realistically 15–20 minutes of tester time per run, so 7–9 tester-hours at minimum, before a single defect is analysed.
- Total capacity: 15 days × about 6 productive hours = 90 hours. The release still needs functional regression, interruption and lifecycle testing, performance and security work.
The stronger argument is return on coverage. The top five combinations cover 63% of sessions; the remaining 23 cover 37%, and include the tablet line at 1.4%. Spending the budget on those 23 buys 37% of usage at the cost of the depth that finds the defects actually generating crash reports.
Offer instead:
- Full functional and exploratory testing on 6 combinations covering ~75%.
- A scripted 10-minute smoke on the top 12, which fits the 300 cloud minutes.
- A named list of excluded combinations with session share and rationale.
- Residual-risk acceptance signed by the product owner.
- A post-release canary: monitor crash-free sessions by OEM and OS for the first 72 hours, and target a patch if an excluded combination spikes.
This meets the intent behind the request - no unknown-unknowns in the long tail - while keeping depth where the risk is concentrated.
3. Cheque deposit uses the camera and biometric login uses the fingerprint sensor. The cloud farm cannot reliably exercise these. How do you cover both features, and what residual risk remains after testing?
Read the worked answer
Camera and biometric are hardware- and OS-mediated. A cloud farm virtualises the sensor, so results there are not evidence of anything.
Camera / cheque deposit, on physical devices:
- Permission states: granted, denied, 'don't ask again', and revoked while the app is in the background.
- Camera already in use by another app; camera unavailable or failing to initialise.
- Capture conditions: low light, glare on the cheque, angled shot, macro distance, autofocus hunting, motion blur.
- Post-capture: crop/deskew, OCR failure path, retry, upload while offline.
- Both a crash-concentrated OEM and an iPhone, because camera HALs and image pipelines differ.
Biometric:
- Not enrolled; enrolled; enrolment changed after app setup.
- Lockout after N failed attempts; fallback to PIN/passcode; user cancels mid-prompt.
- iOS Face ID vs Touch ID differences, and biometric APIs that return 'unavailable' rather than 'fail'.
- Interruption during the prompt (incoming call) and app backgrounded mid-authentication.
Residual risk to document: OEM-specific camera behaviour on devices you do not own, and biometric API behaviour on older OS versions in the excluded matrix. Mitigate with a beta cohort on the excluded OEMs and by watching crash analytics on the biometric and camera code paths during the first release week.