The Nursing Shortlist That Agreed With the Past
A regional university trials an AI system to shortlist applicants for its nursing programme. The vendor reports 91% agreement with human admissions decisions on last year's 2,400 applications. The training data came from five years of decisions in which 78% of offers went to applicants from fee-paying schools, although only 31% of applicants attended them. The system also gives lower scores to applicants who took a gap year for caring responsibilities. The university must decide whether to use it for the coming intake of 600 applicants; the admissions team is under pressure to cut 200 hours of manual review. A regulator requires that automated decisions affecting individuals be explainable and contestable.
1. Why is 91% agreement with past decisions not evidence that the system is fair? Use the figures in the case.
Read the worked answer
Agreement measures imitation, not fairness. The system was trained on five years of human decisions, so 91% agreement means it reproduces those decisions - including their pattern.
The figures show the pattern: fee-paying applicants were 31% of applicants but 78% of offers. A model that learns from those labels will treat school type as predictive and may amplify the gap, because it optimises for the historical outcome. The gap-year penalty is a second problem: caring responsibilities are not randomly distributed and can proxy for sex, disability or income. Even if gender and ethnicity are removed, those proxies remain.
A fair test would report offer rates by group, error rates for successful and unsuccessful applicants, and the effect of removing school type and gap-year explanations. 91% is a starting point for audit, not a certificate.
2. What would a proportionate response be: reject, deploy, or pilot with controls? Justify with conditions.
Read the worked answer
Pilot with controls, not deploy as is. Rejecting outright is defensible but may be premature if the vendor can retrain without school type and gap-year penalties; deploying now would bake in the disparity.
Conditions:
- Remove school type and any proxy, and require the vendor to explain the features used.
- Re-audit on recent data with offer rates by group and a four-fifths-type check.
- Keep a human decision-maker for every shortlist, with written reasons an applicant can receive.
- Publish an appeal route and a contact for contesting a decision, meeting the regulator's explainability requirement.
- Run a shadow mode for one intake before any time saving is claimed.
- Monitor drift and stop if disparities exceed agreed thresholds.
The 200 hours saved is real but not a reason to skip the impact assessment.
3. A manager says 'we can't be biased - the system never sees the applicant's gender or ethnicity.' Explain the flaw.
Read the worked answer
Removing gender and ethnicity does not remove bias; it hides it.
- Proxies: school type, postcode, gap-year reason, subject choices and even essay style correlate with protected characteristics. The model can reconstruct the missing variable from them.
- Label bias: the target is a human decision that already favoured fee-paying schools. The model learns that preference and applies it consistently at scale.
- Measurement: if you never collect group-level outcomes, you cannot see the disparity. Blindness is not fairness.
The correct response is to measure outcomes by group, test for disparate impact, document the features and keep human review. 'The system never sees ethnicity' answers the wrong question.