The Bias Audit That Arrived Late
Global retailer Meridian employs 30,000 people and receives 200,000 applications a year. Since last year it has used an AI screening tool to rank candidates for hourly roles in the US, UK, and Germany. The vendor supplied a US bias audit showing a 0.92 impact ratio for gender, but it did not collect race or ethnicity because the vendor says that is 'not applicable.' The company is subject to New York City Local Law 144 for its NYC stores and to the EU AI Act's high-risk employment requirements as they phase in. The German works council has demanded to know how the tool works and has threatened to block its use. Internal audit found that recruiters cannot explain which candidate features drive the score and that rejection letters are generic. The vendor contract has no audit rights and no obligation to provide feature documentation. HR wants to expand to 12 more countries in four weeks.
1. Identify the compliance and ethical gaps and decide what must pause before the 12-country expansion.
Read the worked answer
Three categories of gap. Legal: NYC Local Law 144 requires an annual independent bias audit, a published summary of the audit and data distribution, and candidate notice; a US gender-only audit with a 0.92 impact ratio does not cover race/ethnicity and may not satisfy the audit requirement. The EU AI Act treats employment screening as high-risk; as a deployer, the retailer must use the system according to instructions, assign competent human oversight, keep logs, monitor, and inform affected workers and candidates. The German works council has co-determination rights over monitoring technology. Contractual: no audit rights and no feature documentation are fatal to deployer obligations. Ethical: recruiters cannot explain drivers, and generic rejection letters undermine the right to reasons.
Decision: pause expansion to the 12 new countries. In NYC, do not process new candidates until a compliant audit is in hand; if the current audit is deficient, suspend NYC use or switch to a rules-based screen. In Germany, do not go live without a works agreement. Existing use elsewhere can continue only with human review of every rejection until documentation and adverse-action letters are fixed. The four-week expansion timeline is not achievable and should be reset in writing to the executive sponsor.
2. Design the vendor negotiation: what evidence and contractual rights you need, and what to do if the vendor refuses.
Read the worked answer
Ask for: the model card and intended use; feature list and feature importance at the segment level; training data provenance and whether protected classes or proxies are included; the full bias audit methodology, including selection rates by race/ethnicity, gender, age, disability, and intersectional groups where lawful; adverse-action reason codes; logs and retention; change notification; and validation on our applicant population.
Contract terms: audit rights, including the right to commission an independent audit and to access data and personnel; documentation delivery obligations; performance and fairness service levels; local validation rights; notification and approval of model changes; data-use restrictions; cooperation with works councils and regulators; termination and transition assistance.
If the vendor refuses, do not expand. Options: renegotiate with a credible alternative vendor that provides documentation, or replace the AI ranking with a rules-based or structured-interview process for the expansion countries. Do not accept a vendor's 'proprietary' claim as a reason to forgo deployer duties; the retailer carries the regulatory and reputational risk. Escalate the refusal to the executive risk committee and document the decision. A short-term delay is cheaper than a regulatory finding or a works council dispute.
3. Define human oversight and candidate communication, including works council engagement.
Read the worked answer
Human oversight design: no auto-rejection. A recruiter reviews every candidate below the interview threshold, sees the top factors in plain language, and can override with a reason. Candidates receive a specific explanation of the outcome and an appeal route. Log every score, view, override, and decision for at least the statutory period. Run quarterly adverse-impact monitoring by role, country, and protected group, with a trigger for review if the impact ratio falls below 0.80. Train recruiters on how to read the score, when to override, and how to avoid automation bias.
Works council engagement: provide a DPIA, the model documentation, the human-oversight protocol, the audit results, and a pilot plan with monitoring; negotiate a works agreement before go-live. Be honest about what the tool does and does not do; do not describe it as 'objective' if the features are not explainable. For NYC, publish the required summary and give candidates notice. For the EU, prepare the technical documentation and logging needed for deployer obligations, and ensure a named human oversight owner.
If the works council blocks the tool, respect the process and use the non-AI process in Germany; that is a legitimate outcome, not a failure of the project. The test of oversight is whether a recruiter can actually change a decision and explain why.