Vendor CV Ranking in a High-Risk Employment Context
A European retailer receives 45,000 applications per year for hourly store roles. It licenses a vendor's CV-ranking model that outputs a 0-100 fit score. The vendor will not disclose model features. The retailer uses the score to shortlist the top 30% for interviews. An internal audit finds the shortlist is 61% male while the applicant pool is 48% male; the gap is largest for candidates over 45. Available fields include name, address, employment gaps, and a free-text availability note. The retailer operates in a jurisdiction covered by the EU AI Act and has a works council that must be consulted. The vendor contract has no audit clause, and the model has been live for 14 months with no monitoring. HR wants to expand to promotions next quarter.
1. How should the retailer classify and govern this system? Explain the reasoning rather than reciting a label.
Read the worked answer
The use is likely high-risk under the EU AI Act because it is an AI system used in employment to select or rank natural persons, specifically shortlisting candidates for jobs. That classification is not a formality. It changes the retailer's obligations as a deployer: it must use the system in accordance with instructions, ensure human oversight, monitor operation, keep logs where required, and ensure input data are relevant and sufficiently representative. The provider also carries obligations for the high-risk system, but the retailer cannot contract them all away.
The governance problem is compounded by the vendor's refusal to disclose features. Without model documentation, the retailer cannot verify data governance, bias testing, or robustness. The works council must be consulted before deployment or material expansion. A data protection impact assessment is likely needed given profiling of applicants. The retailer should treat the system as high-risk, suspend expansion, obtain the vendor's technical documentation and intended-use statement, and establish a documented human-review process in which no candidate is rejected solely by the score. If the vendor will not provide enough information, the retailer should switch to a system it can audit.
2. Diagnose the discrimination risk. What immediate mitigation and validation would you require?
Read the worked answer
The shortlist disparity, 61% male versus a 48% male applicant pool, is a red flag, not proof by itself of unlawful discrimination. The differential must be tested across age, gender, disability, and ethnicity where data are lawfully available, with confidence intervals and by job family. The available fields create obvious proxy risk: address can proxy ethnicity or socioeconomic status, employment gaps can proxy age or caregiving, and free-text availability notes can encode gender or language. Name is rarely job-related and should not be used.
Immediate mitigation: pause score-based shortlisting or require human review of every candidate before rejection; remove name, address, and free text until each is validated as job-related and non-proxy; and test a structured, job-related screening process as a challenger. Do not simply delete gender; proxies can reconstruct it.
Validation: obtain the vendor's training data description and bias test results, then run independent adverse-impact analysis on recent applicants, including selection rates by group and an analysis of score calibration. Test for label bias, such as using past hires as a positive label when past hiring was itself skewed. Require explainability at the feature-group level and a monitoring plan reviewed by the works council and legal.
3. What contractual and operating controls are needed before any expansion to promotions?
Read the worked answer
Expansion to promotions increases risk because it affects existing employees' pay, status, and working conditions. Before that, the contract should include audit and testing rights, model documentation, notification of material model changes, data protection terms, bias-testing evidence, incident notification, and termination or exit rights. The retailer should own or have access to evaluation data and be able to reproduce vendor performance claims.
Operationally, require a documented human decision-maker, candidate notice and contestability, retention limits, access controls, and periodic adverse-impact testing. Set go/no-go criteria such as no unexplained disparate selection rate above an agreed threshold, stable calibration across groups, and completion of works council consultation. Until the current shortlisting use is remediated and monitored, promotions expansion should not proceed. The largest governance failure is not the vendor's black box alone; it is the retailer's lack of audit rights and 14 months without monitoring.