Vendor Resume Screener with Adverse Impact
A staffing vendor offers a resume-screening model that predicts whether a candidate will be invited to interview, trained on five years of a client's human decisions and marketed at 92% accuracy. In a validation sample of 12,000 applications, the selection rate is 0.42 for male candidates and 0.29 for female candidates. Gender is not an input feature; the model uses zip code, employment gaps, university, job-title tenure, and commute distance. SHAP analysis lists employment gap and zip code among the top four features. The client wants to go live in three weeks for 12,000 applications per quarter. Hiring managers would see only the top 100 ranked candidates per role, and no rejected candidate would be told a model was used. The vendor will not share training data or code, but will provide an API and a summary model card.
1. The vendor says gender is not a feature. Does that resolve the adverse impact finding? Explain.
Read the worked answer
No. Excluding a protected attribute does not prevent proxy discrimination. Zip code, employment gap, university, and commute distance can correlate strongly with gender and with other protected characteristics, and a model trained on historical human decisions will learn whatever patterns those decisions contained, including biased ones.
The selection rates give a ratio of 0.29 / 0.42 ≈ 0.69, below the 0.80 four-fifths guideline, which is a signal that requires investigation, not a definitive legal conclusion. The 92% accuracy figure is measured against historical decisions, so it partly measures reproduction of past human behavior rather than merit.
I would require subgroup confusion matrices, impact ratios with confidence intervals, and counterfactual or proxy tests - for example, changing zip code while holding qualifications constant and measuring score movement. The burden is on the vendor and client to show the features are job-related and that less discriminatory alternatives were considered.
2. What would you require before any production use, and what deployment pattern would you accept?
Read the worked answer
Before production I would require access to training data or audit rights, disaggregated performance and selection-rate reporting, a feature-by-feature justification, and evidence that proxy features were tested and remediated. The vendor's refusal to share data or code is itself a governance risk; without auditability the client cannot defend the system to a regulator or a candidate.
I would not accept automated rejection or a top-100 shortlist that silently excludes everyone else. An acceptable pattern is decision support: the model ranks or flags candidates for human review, hiring managers see the full applicant pool, and no candidate is rejected on model output alone. Candidates should be told that automated tools are used, and there should be a human reconsideration and appeal path with adverse-action reasons.
Given the three-week deadline, I would recommend a shadow pilot on one role family, with the model's rankings compared against human decisions and no effect on candidates, while the audit proceeds. Launching at full scale in three weeks is not defensible on the current evidence.
3. Which documentation and monitoring obligations should be in place, and who owns them?
Read the worked answer
Documentation should include an expanded model card covering intended use, training data provenance, features and their job-relatedness, evaluation results by gender and other subgroups, known limitations, and the human-review design. The client should maintain its own AI system inventory entry, a DPIA or equivalent privacy assessment, and a record of the vendor due-diligence review.
Monitoring should cover monthly selection rates and impact ratios by subgroup, score distributions, drift, override rates, complaint volume, and the reasons hiring managers override the model. Any vendor model update should trigger revalidation before it affects candidates. Decision records should be retained long enough to support audits and candidate challenges.
Ownership: the business owner is accountable for use and outcomes; the ML or analytics team owns metrics and monitoring; HR compliance owns process and candidate communications; legal and the DPO own the assessment and contract terms; the vendor owns model documentation, change notification, and audit cooperation. These obligations should be written into the contract, not left to the model card.