Adverse Impact in an HR Screening Model
A staffing firm uses a third-party ML model to rank applicants for warehouse supervisor roles. The vendor reports 92% overall accuracy and a 'bias-tested' claim. Internal audit sampled 4,000 applicants from the last year and found the selection rate for candidates aged 50+ was 18%, versus 22% for candidates under 30, an adverse impact ratio of 0.81. The model uses graduation year as a feature, and the vendor will not remove it because of contract terms. The firm has 300 hires per year and a risk appetite statement that prohibits 'material adverse impact' but does not define the threshold. Legal says the model is not a selection device under current guidance, but HR wants to expand it to 5,000 applications. The risk manager must recommend treatment and whether to expand.
1. Assess the risk significance of the 0.81 adverse impact ratio.
Read the worked answer
0.81 is below the conventional four-fifths threshold of 0.80, so it is a credible red flag even though it is not proof of unlawful discrimination. The risk significance comes from the combination: a protected-class selection-rate gap, a feature that is a close proxy for age, a vendor that will not remove it, and a plan to expand from 4,000 to 5,000 applications.
Overall accuracy of 92% does not address fairness; a model can be accurate in aggregate and still systematically disadvantage a group. Legal's view that the model is not a 'selection device' under current guidance is not a risk acceptance. Guidance can change, and reputational harm does not depend on technical classification. The risk manager should treat this as a high inherent risk requiring validation, not as a settled legal question. The 0.81 should trigger deeper analysis of statistical significance, job-relatedness, alternative features, and whether human review actually corrects the pattern.
2. Recommend risk treatment options and a decision on expansion.
Read the worked answer
Do not expand while the ratio is 0.81 and the proxy feature remains. Options include: avoid the vendor model for this decision or replace it with a simpler, explainable scoring rubric; require removal of graduation year and any age proxies, with contractual audit rights; if the vendor refuses, run a local bias audit on the vendor's scores and add structured human review with criteria blind to age; or reduce reliance by using the model only to prioritize applications for human review, not to reject.
Human review is not a cure if reviewers see the same proxy-correlated scores. The recommended path is a 90-day remediation gate: independent validation, feature review, adverse impact analysis, and a documented decision by the risk committee. Expansion should be denied until the ratio is at least 0.90 with statistical confidence and no unexplained gap, or until the firm accepts residual risk at the executive level with a clear rationale. Because the risk appetite statement is undefined, the committee must set the threshold now.
3. If HR insists on continued use, what residual risk acceptance and monitoring should the risk manager require?
Read the worked answer
Residual acceptance must be explicit, time-bound, and owned by an executive, not by HR alone. Conditions:
- Define 'material adverse impact' in the risk appetite statement, for example an adverse impact ratio below 0.90 triggers remediation.
- Monitor quarterly by age and other protected classes, with selection rates, score distributions, override rates, and complaint data.
- Require an independent audit annually and after any model or feature change.
- Amend the vendor contract for audit, feature removal, and notification of model changes.
- Provide adverse action notices and genuinely independent human review.
- Set a stop-use trigger if the ratio falls below 0.85 or a complaint pattern emerges.
The risk committee should document why continued use is acceptable, what residual risk remains, and what would cause withdrawal. If the firm cannot meet those conditions, the correct decision is to pause the tool rather than rely on disclaimers.