Risk-Based Usability Test Planning for a Clinical Appointment Scheduler
A hospital is releasing a patient-facing appointment scheduler in six weeks. Users include older adults and people with low digital literacy. A prior pilot incident: a patient booked at the wrong clinic because the scheduler defaulted to the last-used location and did not show the clinic name on the confirmation screen. The team has budget for eight usability test sessions and one accessibility review. Clinical safety staff want assurance that scheduling errors are reduced. The product owner wants a quick "smoke test" with five staff members. The scheduler includes a three-step wizard, a location search, and a confirmation screen. The hospital's quality process references ISO 9241-11 for usability and ISO 14971 for risk management. The release cannot slip. The test manager must define usability risks, select a test approach, and set exit criteria.
1. Identify the main usability risks and rank them by severity and likelihood. Explain how the prior incident informs the ranking.
Read the worked answer
Main usability risks:
- Wrong clinic selection: a patient books at an unintended location because the default is persisted or the clinic name is not prominent. Severity is high because it can cause missed appointments and delayed care; likelihood is high given the prior incident.
- Confirmation does not show critical details: patient cannot verify clinic, date, time, or provider. High severity; likelihood is high if the screen emphasises "success" over the booking details.
- Older adults or low-digital-literacy users cannot recover from errors: they abandon or call for help. Medium-to-high severity; likelihood is elevated by the target population.
- Wizard navigation loses entered data: duplication and frustration. Medium severity; likelihood depends on implementation.
- Accessibility barriers: screen reader or keyboard users cannot complete scheduling. High severity for affected users.
Rank using severity and likelihood, not just frequency. The prior incident is direct evidence that the wrong-clinic hazard is credible and not hypothetical. It should be treated as a known hazard-related use scenario, not as an isolated complaint.
2. Choose the test participants and method. Why is testing with five staff members inadequate?
Read the worked answer
Test with representative patients, not only staff. Recruit eight participants across the target population: older adults, people with low digital literacy, at least one assistive-technology user, and a mix of prior online booking experience. Use moderated, task-based sessions with realistic appointment scenarios, including a scenario that requires choosing a different clinic from the default. Add the accessibility review with keyboard and screen reader checks.
Five staff members are inadequate because staff have expert knowledge of the system and the organisation. They may know clinic names, defaults, and workarounds that patients do not. They are also not representative of the target users' digital literacy or accessibility needs. Staff can provide heuristic feedback, but they cannot replace patient testing for a patient-facing safety-related workflow. A smoke test may find obvious defects, but it is not sufficient evidence for the wrong-clinic risk.
3. Define exit criteria that connect usability evidence to risk control. Which standards would you cite and how?
Read the worked answer
Exit criteria should be risk-based and evidence-based:
- No participant completes a booking at the wrong clinic in the wrong-clinic scenario.
- Every participant can state the clinic, date, and time from the confirmation screen without prompting.
- No critical usability defect remains uncontrolled for the wrong-clinic or confirmation hazards.
- Accessibility review finds no blocker for keyboard or screen reader completion of the core task.
- Any residual risk is documented and accepted by clinical safety.
Cite ISO 9241-11 to define usability in terms of effectiveness, efficiency, and satisfaction for the specified users and context of use. Cite ISO 14971 to connect usability findings to hazard identification, risk estimation, risk control, and residual-risk evaluation. The standards do not replace testing; they structure the evidence. The eight sessions are qualitative, so the team should not claim statistical prevalence. Instead, the evidence should show that the known hazard-related scenario has been exercised and that risk controls are verified.