Mobile device test matrix: how to choose what to test

A mobile device test matrix tells your team which phones and OS versions to test, what to check on each, and which app build the results apply to. Choose devices based on who uses your app and what could break in this release. Then give each check an owner and somewhere to record the result.

Below is a template you can copy, followed by an example for a release that changes photo uploads. For help recruiting testers, distributing builds and planning the rest of the beta, use the mobile beta testing checklist.

What should a mobile device test matrix include?

Give each row a specific job. “Test sign-in on this phone” is a start, but “sign in after upgrading from the previous build, with an expired session” tells the tester what to do. Fresh installs and upgrades need separate checks, even on the same phone.

Copy this into your test document or test-management tool:

Release candidate: [version, build and distribution channel]
Test environment: [backend, feature flags and test-data set]
Supported devices: [platforms, OS versions, phones and tablets]

Row ID:
Device model / physical, emulator or simulator:
Installed OS version:
Why this combination matters:
What to test and what should happen:
Setup: [fresh install / upgrade from build / account / permissions]
Settings to use: [locale, text size, orientation or network condition]
Tester:
Build actually tested:
Result: [not run / passed / failed / blocked]
Evidence or issue link:
Anything left to check:

Fill in the exact versions when you run the test. “Latest iOS” stops being useful after the next update. Record the build you actually installed, too: yesterday's passing result may belong to a different build from the one you're about to release.

How do you choose which devices and OS versions to test?

Look at what you support, what your users have and what changed in the release.

• What you support: Include your oldest supported OS versions and any tablets or foldables you support. A pass on a new phone won't tell you whether the app still works at that lower limit.
• What your users have: Check your app analytics, support reports and customer commitments. If the app is new, start with the devices you expect testers to use and revisit the choice after the beta.
• What changed: A new camera flow needs different checks from a translated payment form. Pick devices and settings that let you test those changes properly.

The most popular phone in your analytics is a sensible place to start. You may also need an older phone used by a particular customer, or the model where an earlier release broke. Write a short reason beside each choice; it makes the list easier to review next time.

Check that you can run the selected OS on the selected model. If you use a hosted lab, confirm the combination is available before assigning it. Firebase documents its available-device lookup and notes that tests on removed devices are skipped. A skipped test still needs doing elsewhere or a decision to release without it.

What could a small release matrix look like?

Suppose you're testing a field-service app that has changed photo uploads and its job form. The example below shows how you might divide up the work. Replace the device descriptions with exact models and OS versions before handing it to testers.

ID	Device to use	What to test	Why include it?
A	Android phone and OS version common among your users	Upgrade from the previous production build with a queued photo; resume and retry	Check whether a queued upload survives the update
B	Supported Android device with limited memory	Take a job photo, leave the app, then return to the unfinished form	Check whether the form keeps unsaved work
C	iPhone running your oldest supported iOS version	Fresh install; deny photo access, then follow the recovery path	Check that people can recover after denying access
D	iPhone and iOS version common among your users	Edit a populated job form with larger text and the keyboard open	Check that the keyboard doesn't hide required controls
E	Supported tablet, if the app serves tablet users	Open and edit the job form in supported orientations	Catch layout problems on tablets

If row A fails, try to reproduce it with the same queued photo and upgrade path. Reinstalling the app may remove the very condition that caused the bug. If nobody has a tablet for row E, leave it marked as not run and raise it at the release review.

How do you stop the matrix becoming too large?

Run the app's essential flows on your main test devices, then add checks for changed features and bugs you've fixed before. You don't need to repeat every check in every setting. If you've changed how the app handles decimal separators, test amount entry in the affected locales; repeating the camera check in each language may add little.

Some combinations do matter. An upgrade test and an offline-upload test won't cover upgrading while an upload is already queued. Add a row for that case if users can reach it.

Keep smaller automated tests alongside the device checks. Android's testing strategy guidance distinguishes unit, component, feature, application and release-candidate tests. A unit test can check an amount calculation without running the whole app. Use the device test to check that a person can enter the amount and finish the payment.

When should you use physical devices, emulators or simulators?

Simulators and emulators make it easier to repeat tests across screen sizes and OS versions. Physical devices let you check how the app behaves on the hardware people use. Mark which kind you used in the matrix.

Apple explains that simulated devices do not replicate a physical device's performance or features. For the photo-upload example, take a photo on a real phone and follow it through to upload. You can use a simulator to check the job form's layout, but that won't cover the camera itself.

If you use an automated device lab, link its results back to the plan. Firebase Test Lab uses “test matrix” for a batch of executions identified by dimensions including model, OS version, orientation and locale. Its matrix documentation explains those executions. Keep your worksheet's setup instructions and tester names alongside the results, so someone can follow up on a failure or rerun a check.

How do you connect failed checks to useful bug reports?

When a check fails, link the row to a bug report. Include what happened, what should have happened, the steps, the build and any setup needed to reproduce it. Add the row ID to the report so the tester can return to the same check once it's fixed. The beta issue template gives you a format to copy.

If a tester can't sign in because their test account has expired, mark the check as blocked and get the account fixed. Reserve “failed” for a problem with the app.

BugScreen's embedded SDKs attach device model, OS and app-version context to submitted reports; the Android and iOS collected-data guides describe the fields. The tester still needs to explain what they did before the failure. If they use the separate companion app, ask them to include the tested app's build number: the companion cannot read another app's build metadata or logs.

What should you review before signing off the release?

Check that the results apply to the build you're releasing. Open the linked bugs, confirm the fixes have been retested and follow up on blocked checks. If another build arrives, decide which tests to rerun and keep the earlier results labelled with their original build number.

Go through anything left untested with the person making the release decision. Record what they accept, why, and who will follow up. That gives the next QA round a useful place to start.

If chasing screenshots and build numbers is taking time away from testing, explore BugScreen's QA reporting workflow or create a workspace to try it with your testers.