Choosing a Formative Usability Method Under a Three-Week Deadline
A B2B expense-reporting vendor has three weeks before releasing a redesigned admin console. A quarterly survey of 240 administrators found that 38% rated "finding and fixing a rejected expense" as difficult, but the survey offered no diagnostic detail. Support tickets mention "rejected" and "resubmit" but are not reproducible from logs. The release adds an approval dashboard and changes the rejection workflow. The team has one researcher, no permanent lab, and budget for eight participant incentives. The product manager proposes a 12-participant unmoderated remote test with recorded task attempts; engineering prefers six moderated remote sessions; sales offers guerrilla testing at a conference with walk-up participants. The researcher must choose an approach that produces actionable findings in two weeks, while the release scope is still changeable. Observers from compliance want to watch live, and the team wants a metric it can compare with the previous release.
1. Which usability testing approach should the researcher choose, and how should participants be sampled? Justify the method against the decision deadline, diagnostic goal, and observer constraints.
Read the worked answer
The researcher should choose moderated remote sessions with six to eight representative administrators, not the 12-participant unmoderated test and not conference guerrilla testing.
Reasoning:
- The goal is diagnostic and formative: the survey says users struggle, but not why. Moderated sessions allow follow-up questions, observation of recovery behaviour, and probing of confusing labels or workflow steps.
- Unmoderated testing scales better, but with one researcher, 12 sessions create analysis load and give no live probing. Recorded task attempts may show failure but not the user's interpretation.
- Guerrilla testing at a conference samples the wrong population. Administrators who process expense rejections have domain knowledge and permission structures that walk-up participants will not share.
- Six to eight participants are appropriate for qualitative formative testing: the aim is to find severe, recurring problems, not to produce statistically generalisable prevalence estimates.
Sampling should be purposive and screened: current administrators who have handled at least one rejected expense in the last 30 days, stratified by organisation size and rejection frequency. Include a mix of frequent and occasional users. Compliance observers can watch remotely with consent and use a structured note template; they must not coach participants. A pilot session should test the script, recording, and remote connection before the real sessions.
2. Design task scenarios and success criteria for the rejection workflow. Explain why you would not ask participants whether they "like" the new dashboard as the primary measure.
Read the worked answer
A task scenario should give a realistic goal without naming the interface controls. For example: "A colleague's expense claim was rejected because the receipt is missing. Find the claim, correct the problem, and resubmit it so it can be approved." Start the participant from a realistic inbox or notification state.
Measure:
- Completion: claim corrected and resubmitted without moderator assistance.
- Critical errors: resubmission to the wrong approver, deletion of the claim, or repeated failure to find the rejection reason.
- Assistance: number and type of moderator prompts.
- Time and navigation path: useful diagnostically, but not the sole success criterion.
- Comprehension: after the task, ask the participant to explain what happened and what happens next.
Do not lead with "Do you like the new dashboard?" Preference is weak evidence and invites politeness bias. Behavioural evidence should come first; satisfaction can be collected afterwards with a short validated instrument or open debrief. If the team needs comparability with the previous release, use the same post-task questions and the same core task, but treat the comparison as indicative because sample sizes are small. A pilot participant should be used to check that the scenario is unambiguous.