Two Out of Five: Classifying a Blanket Exclusion from Impact Assessment
You are conducting the stage 2 audit of a retail bank's AIMS. Five AI systems fall within the scope. AI system impact assessments exist for three; the missing two are a résumé-screening model used in the bank's own recruitment and a sentiment-analysis model applied to free-text employee survey comments. The AI governance manager states that the bank's methodology, approved by the AI steering committee, holds that impact assessments are required only for customer-facing systems, since internal systems "do not process customers". Separately, model retraining occurred 12 times in the audit period and the required retraining record exists for 10; the manager says the two gaps were operator error and the operators have been reminded. You also saw a human-oversight dashboard with alert thresholds and a documented escalation path, which the manager wants recorded as a strength. Discussions are becoming tense.
1. How do you decide whether the two missing impact assessments constitute a nonconformity, and how would you grade it?
Read the worked answer
Test the requirement against the facts, not the organisation's methodology. The standard requires the organisation to establish and apply an AI system impact assessment process for its AI systems, addressing impacts on individuals and groups. It contains no customer-facing carve-out. The bank's criterion is therefore not a permissible tailoring decision; it is a written rule that excludes systems from a required process.
Before concluding, check for equivalence: did the bank perform an alternative assessment of comparable depth for these two systems - a data-protection impact assessment, a recruitment-bias review, an employee-consultation record? Read them. If an equivalent assessment exists and addresses the required elements, the finding may reduce to a documentation and methodology issue rather than a gap in analysis.
Grading. If nothing equivalent exists, this is a major nonconformity: the failure is systemic, arising from an approved methodology rather than an isolated lapse, it affects two of five systems in scope, and both systems can produce adverse effects on individuals (employment decisions, inferred sentiment). It is not one missed document. If an equivalent assessment exists but is not recognised or controlled by the AIMS, a minor nonconformity against the process requirement is more defensible.
Record: the requirement, the objective evidence (methodology, committee approval, the two systems, absence of assessments), the consequence, and the grading rationale.
2. How do you treat the two missing retraining records, and what does that tell you about your sampling?
Read the worked answer
The organisation has a defined retraining process, documented information requirements, and 10 of 12 records produced on request. That pattern - process exists, control operates, isolated lapses - points to a minor nonconformity against the operational control and documented-information requirements, unless further sampling shows the gap is wider.
Sampling response. Do not close the sample at 12. Expand it: pull the full retraining population for the period, including any retraining triggered by model change or incident, and check a second, independently selected set. Also inspect the monitoring that should have detected the missing records before the auditor did - if there is no reconciliation between retraining events and records, that is a control weakness and may shift the grading. Ask what the two missing records would have evidenced: competence of the operator, or approval of the model version? The answer changes severity.
Avoid the trap. The manager's explanation is a claim, not evidence. "Operators were reminded" is a correction; it does not establish why the documentation control failed, nor that it will not recur. Note the explanation and test it against records.
3. The manager disputes the impact-assessment finding and asks you to soften it in exchange for recording the oversight dashboard as a strength. How do you handle the closing meeting?
Read the worked answer
Separate the three issues and refuse the trade.
- The dashboard stands on its own evidence. Good practice may be recorded in the report as an observed strength because it is true, not as a concession. It does not mitigate a nonconformity in a different process; auditors do not net strengths against findings.
- The finding is written against the requirement, with the evidence and the consequence, whatever the client's preference. Softening a finding to preserve the relationship compromises fair presentation and independence.
- The disagreement must be captured, not suppressed: record the client's position and the reason it was not accepted. Many schemes require the audited organisation to be given the opportunity to comment, with the comment retained in the file and the certification decision-maker able to see it.
Process. Present findings in order of significance with the evidence attached; do not debate grading in open session with a group. Offer a separate discussion with the governance manager, then escalate unresolved disagreement through the CB's appeals route. If the atmosphere prevents a productive closing meeting, adjourn, document why, and reconvene. The audit conclusion, not the client's comfort, is the deliverable.