Research, Testing, and User Evaluation with People with Disabilities
Three testing layers
- Automated scanning - fast, repeatable, and shallow. Commonly catches only a fraction of real issues (roughly a third), mostly mechanical ones like missing alt text or contrast.
- Manual expert testing - keyboard-only traversal, focus order, screen reader smoke tests, 200%/400% zoom, reflow at 320 CSS px, text-spacing overrides, reduced motion, forms and error handling.
- Usability testing with people with disabilities - reveals whether tasks can actually be completed.
Running inclusive research
Recruit by disability type and AT used, not a generic "disabled" quota. Match the AT to the participant's own setup where possible. Budget extra time (screen reader tasks run slower), offer materials in advance, use accessible consent and scheduling, allow support persons, compensate fairly, and ask about accommodations at recruitment. Remote sessions need accessible platforms - a captioned, keyboard-operable meeting tool.
Method traps: pure think-aloud overloads screen reader users who are already processing audio; consider retrospective narration or dual-channel prompts. Do not conflate a compliance audit with generative research - different goals, different protocols. One participant's opinion is not validation of a product; small samples reveal problems, not prevalence.
Other traps: "automated tools give us a pass"; "we tested with a screen reader, so we're covered" (magnification, speech input, cognitive load, and motor access are separate); testing only the happy path; ignoring AT version and browser pairing in reporting.
Worked example
A checkout flow passes automated checks. Manual testing shows the error message is not announced; a screen reader participant confirms she did not know the card field was rejected. Fix: move focus to an error summary, associate errors programmatically, and re-test with two participants.
Rule of thumb
Automate what is mechanical, test manually what is behavioral, and involve disabled users to learn what is actually usable.