Ranking jobseekers with an opaque AI tool

Classification: The tool ranks jobseekers for access to training. That is likely a high-risk AI system under the employment/education categories, because it materially influences access to training and employment opportunities. Classification should be confirmed against the AI Act annexes and the vendor's intended-purpose documentation, but the agency should plan on the high-risk assumption.

Deployer obligations (high-risk):

• Use the system in accordance with instructions and ensure human oversight.
• Implement data governance and monitor for bias and drift.
• Keep logs and inform affected persons.
• Conduct a fundamental rights impact assessment if the agency is a public body or the use falls within the relevant scope.
• Register the use where required.
• Ensure staff have sufficient AI literacy.

Practical point: The agency cannot outsource its deployer obligations to the vendor. The vendor's technical documentation is necessary but not sufficient; the agency must be able to explain and challenge the tool's outputs.

Legal basis: The agency needs a clear legal basis, likely a public task or legal obligation, with a specific legal mandate for ranking. 'Consent' is unlikely to be valid in a public-service context with unequal power.

Necessity and proportionality: Are all 40 variables necessary? Postcode may act as a proxy for ethnicity or socio-economic status; 'motivation score' from free text may be unreliable and opaque. The agency should reduce variables to those that are relevant and validated.

Automated decision-making: If the tool makes a decision with legal or similarly significant effects based solely on automated processing, Article 22 GDPR applies. A ranking that effectively excludes people from training may qualify. The agency should ensure meaningful human intervention, the right to express a view, and the right to contest.

DPIA: A DPIA is required because the processing involves profiling, vulnerable data subjects and potentially automated decision-making. It should cover: description, necessity, risks of bias, transparency, retention, security, and mitigations. The DPO must be consulted.

Conclusion: The pilot should not start until the DPIA is complete and the legal basis and human-oversight model are documented.

Trade secrets do not eliminate regulatory access or the agency's accountability. The agency should insist on:

• Technical documentation sufficient to assess purpose, data, model logic, limitations and validation.
• Bias and accuracy testing on the agency's population, with results shared.
• Audit and inspection rights for the agency, DPO, supervisory authority and AI Act market-surveillance authority.
• Human-oversight functionality: explanations of rankings, ability to override, logging.
• Data ownership and return/deletion on exit; no secondary use by the vendor.
• Change control: notification and re-testing before model updates.
• Liability and indemnity for unlawful processing or discriminatory outcomes.
• Exit and portability to avoid lock-in.

If the vendor refuses audit rights or meaningful documentation, the agency should not procure. A six-month pilot should be a controlled evaluation with a stop rule, not a three-year commitment. The agency should also consider a smaller proof-of-concept with an explicit go/no-go decision.