Six Participants, Eighty Percent, and a Launch Decision
A civic tech nonprofit runs a usability test of a new benefits application with 6 participants: 4 recruited through a general panel and 2 through a disability advocacy group. Only 1 uses a screen reader regularly. Findings show 5 of 6 completed the form, 2 of 6 struggled with identity proofing, and one participant's assistive technology crashed the session. The product owner says “80% completion means it is accessible” and wants to launch.
1. Critique the claim that 80% completion demonstrates accessibility.
Read the worked answer
Completion rate alone is not an accessibility measure. With 6 participants, 80% is 5 people; one non-completion is a large proportion and may signal a serious barrier. The test does not report assistance, time on task, errors, or where the 2 identity-proofing failures occurred. A single assistive-technology crash is a critical reliability signal, not an outlier to average away. Accessibility requires that people with diverse disabilities can complete core tasks with their own technology. A headline percentage can hide exactly the participants who experienced barriers.
2. What can and cannot be concluded from this sample composition?
Read the worked answer
The sample is too small and too narrow to generalize about accessibility. Only 1 regular screen-reader user, 2 disability-community recruits, and no described representation of keyboard-only, low-vision, cognitive, or motor users means major interaction modes are untested. You can conclude that at least one assistive-technology crash occurred and that identity proofing caused difficulty for some participants. You cannot conclude the application is broadly accessible, nor that it is inaccessible for all screen-reader users. The findings are directional and should drive targeted follow-up testing.
3. What should happen before launch?
Read the worked answer
Do not launch on the 80% claim. First, investigate the crash and the identity-proofing failures as defects. Reproduce the crash with the same assistive technology and browser/device combination. Run a focused round with more screen-reader, keyboard-only, low-vision, and motor-impaired participants on the identity-proofing and core submission flow. Define pass criteria by task and assistive technology, not by aggregate completion. If launch must proceed, publish known limitations, provide an assisted alternative, and commit to a dated remediation and retest. The product owner should treat the crash as a release risk, not a statistic.