Staffing Agency: Protected Attributes Removed, Disparity Intact

Removing a feature is not removing the signal.

1. Proxies carry the information. University name correlates with socioeconomic background and, in this agency's history, with the roles and clients it has served. A 14-month employment gap encodes caregiving, illness, or redundancy - all of which correlate with protected characteristics. The model can reconstruct much of what was deleted.
2. The model's priors come from the wider world. A hosted model is pretrained on text that encodes historical hiring patterns; free-text CVs and cover letters carry signals in phrasing, school names, and activity descriptions that no field removal touches.
3. The omission changes nothing about the objective. The model is ranking for 'who looks like a strong candidate' as learned from general data, not for 'who will succeed in this role' as measured by this agency's outcomes.

Consequence for the claim. 'We removed protected attributes' is a statement about inputs, and discrimination law and fairness assessment concern outcomes. The audit showing 3.1x and 0.4x is outcome evidence and it contradicts the claim. The honest formulation is: 'We have no evidence the tool's outputs are free of disparate impact; the audit suggests otherwise.'

Evaluation

• Disaggregated selection rates by protected group where data is lawfully held, plus by proxy groups (university tier, employment-gap presence, career-break reasons). Compare each to the overall shortlist rate; a 3.1x ratio is a red flag on any threshold used in adverse-impact screening.
• Counterfactual probing: take real applicant pairs identical except for university name or a date-shifted employment gap, and measure rank change. This isolates the proxy's effect without needing labels.
• Validity evidence: does shortlisting predict anything the agency cares about - interview pass rate, offer acceptance, 12-month performance? If not measured, 'fit for purpose' cannot be asserted; it is currently a claim about efficiency, not effectiveness.

Operational control

• Human decision on every shortlist: recruiters must review beyond the top 40, record their reasons against a published rubric, and be able to add candidates the model omitted. The model becomes an aid; the decision is human and attributable.
• Ongoing monitoring: quarterly disaggregated metrics, a documented threshold that triggers suspension, and a named owner.
• Documentation: a model card covering intended use, known limitations, evaluation results, and the specific populations excluded from validation.

Fitness for purpose must be scoped: the tool may be defensible as 'screening assistance with mandatory human review' and indefensible as 'automatic shortlisting'.

What can honestly be provided now: very little.

• Asking the LLM to explain its own ranking produces a plausible narrative, not a faithful account of the cause. Post-hoc rationalisations from a model are known to be unstable - rephrasing the question yields a different reason. Presenting that to a regulator as the reason would be a misrepresentation.
• What is defensible is system-level disclosure: the criteria in the prompt, the applicant fields used, the model and version, the evaluation results including the disparities found, and the fact that shortlisting is one input to a human decision.
• The recruiter who accepted the top 40 without review has no recorded reason either, so there is no human rationale to fall back on.

What must change

1. Structured rubric scoring against job-related criteria, with the model used for extraction and summarisation of evidence, not for an opaque ordinal rank.
2. Per-applicant evidence trail: which rubric criterion was judged unmet, with the evidence considered. That is an explanation a regulator can test.
3. Recorded reviewer rationale at the point of decision, including departures from the model's ordering.
4. Notice and contestability for candidates, stated in the recruitment process.

The general lesson: explainability has to be designed into the decision record. It cannot be reconstructed after the fact by asking the model why.