Monitoring and Measurement of an AIMS Based on ISO/IEC 42001
Domain 5 covers Clause 9: the evidence that the AIMS works. Items ask what must be monitored, who may conduct an internal audit, and which inputs management review must receive.
9.1 Monitoring, measurement, analysis, evaluation
Determine what to monitor, the methods, when, and who analyses and evaluates; retain results as documented information. Two metric families are needed. AIMS process health: percentage of AI systems with a current impact assessment, treatment actions closed on time, training completion, incident closure time, supplier assessments completed. AI system performance and trustworthiness: drift, accuracy against thresholds, human override rate, fairness indicators, complaint volume. Neither family alone is sufficient.
9.2 Internal audit
Audit at planned intervals against defined criteria and scope; auditors must be objective and impartial, so nobody audits their own work; results are reported to relevant management and retained.
9.3 Management review
Inputs include status of actions from previous reviews, changes in context, performance information (nonconformities, monitoring results, audit results, interested-party feedback, risk assessment status, improvement opportunities), and resource adequacy. Outputs are decisions on improvement and any need to change the AIMS.
Worked example
A team monitors model accuracy monthly. Accuracy stays stable, but six months later a regulator asks for the impact assessment covering a changed use of the system, and none exists. The diagnosis: monitoring covered the AI system's technical performance but not AIMS performance. The fix is to add process indicators and a change-management trigger requiring an impact assessment re-run for material changes to use, data or model class, reported quarterly into management review.
Traps that cost marks
- Collecting data without analysis and evaluation - Clause 9.1 requires all four activities.
- Assigning the internal audit to the AI system owner or the person who built the process.
- Management review minuted as "no issues" with no decisions, resources or follow-up actions.
- Choosing metrics unrelated to objectives and risks, then wondering why they prove nothing.