Five Sessions and a Number in the Exec Deck
Metro Power is testing a prototype of its outage-reporting app. The researcher ran five moderated remote sessions with customers recruited from the call center. Results: four of five participants found the outage map within 30 seconds; two of five completed the report flow; one participant was interrupted mid-session by a delivery and rushed the last two tasks; one participant had never used a map app and did not recognise the pin icon. The product manager now wants to run a survey asking "How easy and useful was the outage map?" on a 1–5 scale, and to quote "40% task success" in the executive deck. A second stakeholder asks why only five people were tested and wants 30 more recruited immediately.
1. What can and cannot be claimed from these five sessions? Address the "40% task success" figure and the two anomalous participants.
Read the worked answer
Five moderated sessions are a formative method. They are excellent at finding problems, because the same few interface issues recur across participants; they are not designed to estimate a rate. Reporting "40% task success" treats two out of five as a population parameter. A 95% confidence interval for 2 of 5 runs from roughly 5% to 85% - wide enough to be useless in an executive deck, and dangerously easy for a stakeholder to quote back as a target.
What you can say: "In five sessions, three participants failed to complete the report flow, all at the same step" is a defensible, actionable finding. Report counts, quotes, and the failure mechanism, not percentages.
The two anomalies are not noise to be discarded:
- The interrupted participant rushed the last two tasks, so their failures should be interpreted cautiously - but they should not be silently deleted, because the pattern across others is what carries the finding.
- The map-illiterate participant is a genuine and commercially important segment: older or lower-digital-literacy customers, plausibly over-represented among people who call rather than use the app. Their confusion about the pin is a finding about iconography, not a bad data point.
Also flag the recruitment bias: sampling from the call center selects for people who already had difficulty with the service. That is useful for finding problems and misleading for estimating prevalence.
2. Rewrite the survey question and explain what is wrong with it. What would you recommend instead?
Read the worked answer
The question - "How easy and useful was the outage map?" on a 1–5 scale - fails in four ways:
- Double-barrelled. Easy and useful are different constructs. A map can be easy to read and useless for finding your outage. A respondent cannot answer both with one number, and you cannot interpret the result.
- Leading and vague. "How easy" presupposes ease; "useful" has no referent - useful for what task?
- Unlabelled anchors. A 1–5 scale without words means each respondent invents their own scale.
- A prototype artefact. Asking about a feature in isolation loses the task context that determines whether it works.
Recommended replacements:
- Single Ease Question, asked immediately after the task: "Overall, how difficult or easy was it to report your outage?" on a 1–7 scale, anchored "Very difficult" to "Very easy".
- A separate usefulness item, asked after the session: "How useful would this feature be for reporting an outage in the future?" with labelled points.
- For overall perception, use a validated instrument such as SUS rather than inventing items.
More importantly: a survey measures prevalence and satisfaction, not causes. If the question is "why did three of five fail?", a survey cannot answer it. Run the survey after the fixes, to check whether the problems found in qualitative testing exist at scale.
3. Should the team iterate and retest with five, or recruit 30? Explain the reasoning and what each option buys.
Read the worked answer
These answer different questions, and the stakeholder is asking a summative question while holding formative evidence.
Iterate and retest with five is right now. The purpose of the next round is to find out whether the fixes work. Most severe usability problems surface within the first few participants, so a second round of five with a new set of participants will tell you whether the same failures recur. The three failures were concentrated at one step, which is a specific, fixable target; thirty people on a design you already know is broken spends the budget measuring the defect precisely rather than removing it.
Thirty participants buys you a defensible success rate with a usable confidence interval, and it is the right approach for a benchmark once the design is stable - ideally within a larger benchmark study, using the same tasks and definitions so results are comparable over time. It is a poor use of effort before the obvious failures are addressed, because the headline number will be bad and you will have to explain it twice.
Practical plan: iterate, retest with five, fix again if needed. Then run a benchmark with a larger sample - or, cheaper and more representative, instrument the live app and measure task completion in production. Before either, agree with the PM on what metric is being reported and how it will be described, so a count is never dressed up as a percentage.