Generative AI Recruiting Assistant With Adverse Impact and Vendor Opacity
A European retailer plans to deploy a vendor-hosted generative AI assistant to help 60 recruiters screen 25,000 applications per year. The assistant summarizes CVs, ranks candidates, and drafts interview questions. The vendor contract permits model updates with 10 days' notice; a new version was released last month. A pilot on 500 applications found that the ranking model recommended 31% of male applicants but 19% of female applicants for interview, an adverse-impact ratio of 0.61 against the retailer's 0.80 threshold. Recruiters accepted hallucinated qualifications in 6 of 40 audited summaries. The vendor will not provide training data or model weights, but offers a bias audit report and an API. The retailer's AI policy requires a risk assessment before go-live, and the works council has demanded human review of every rejection. The retailer operates in three EU countries and has already received two candidate complaints about automated screening.
1. Conduct the risk assessment the policy requires. Identify the principal risks, their likelihood and impact, and the controls that would change the residual rating.
Read the worked answer
The principal risk is discriminatory impact in hiring. The pilot adverse-impact ratio of 0.61 is below the retailer's 0.80 threshold; with 25,000 applications, a 31% versus 19% interview recommendation rate could affect hundreds of candidates and expose the retailer to employment-law claims and reputational harm. Likelihood is high because ranking is the core function and the vendor update already changed behavior; impact is high for candidates and the business. The second risk is unreliable output: 6 of 40 summaries, or 15%, contained hallucinated qualifications, which could lead to wrong interview decisions. Third is vendor opacity and uncontrolled model change: no weights or training data, 10-day notice, and a new version already in use. Privacy risk is moderate if CV data are processed lawfully, but retention and secondary training use need contract terms. Inherent risk is high. Controls that reduce residual risk: disable automated ranking; use the tool only to summarize with citations; human review of every rejection; local bias testing on at least 1,000 applications; contractual change control and audit rights; recruiter training; and monitoring. Residual risk is medium only if ranking is removed or bias is demonstrated below threshold. Without those controls, residual risk remains high and deployment should not proceed.
2. Recommend a go/no-go decision with conditions, and state the gate metrics that must be met before expanding.
Read the worked answer
Do not approve unconditional go-live. Approve a limited pilot for summarization and interview-question drafting only, with automated ranking disabled. Conditions: (1) the vendor provides a version-specific bias audit for the current model and commits to 30 days' notice and no silent updates; (2) the retailer runs a local validation on at least 1,000 recent applications, stratified by gender and other protected groups, and shows an adverse-impact ratio of at least 0.80 with confidence intervals; (3) every rejection and every ranking change has documented human review; (4) hallucination rate in audited summaries is below 2%, with citations to the CV and a recruiter correction log; (5) a data protection impact assessment and works council agreement are completed; (6) candidate complaints and override rates are monitored monthly. Gate metrics for expansion: adverse-impact ratio at least 0.80 for two consecutive quarters, hallucination below 2%, human override rate reviewed for patterns, no material privacy incident, and vendor change notices tested. If the vendor will not support these, limit use to non-decision support or select another vendor.
3. What contractual and assurance controls would you require from the vendor, given that it will not disclose training data or model weights?
Read the worked answer
Because the model is opaque, rely on output-level assurance and strong contract terms. Require version pinning and a change log; at least 30 days' notice before any model update and a right to reject an update that fails local bias tests; audit rights and access to bias audit methodology; test data and API access for independent evaluation; prohibitions on using the retailer's candidate data to train the vendor model or for any secondary purpose; data retention and deletion terms; security certifications and breach notification; service levels for availability and latency; liability and indemnity for discrimination and privacy claims; and exit assistance, including deletion and transition. Add a right to conduct black-box bias testing and to suspend the service if adverse impact exceeds threshold. Do not accept a generic bias report that covers a different model version. If the vendor refuses audit rights or change control, treat that as a residual risk requiring a lower-risk use case, not as a reason to skip the control.