Incident Response and Assurance
Incident response and assurance
This domain tests what you do when an AI system misbehaves, and how you prove afterwards that controls work. It joins AI-specific response actions with the assurance evidence auditors and regulators expect.
Core rules
- Follow the standard lifecycle - prepare, detect and analyse, contain, eradicate, recover, post-incident review - with AI-specific playbooks for poisoning, model theft, harmful output, prompt-injection exfiltration and model outage.
- Containment options unique to AI: roll back to a known-good model version, disable tools or agency, raise output filtering, quarantine an index or data source, rotate keys, and revoke endpoint access.
- Preserve evidence: request/response logs, model version, retrieved documents, tool-call traces, registry history. Snapshot before remediation.
- Notification duties may apply: GDPR 72-hour breach notification for personal-data incidents, and serious-incident reporting for high-risk systems under the EU AI Act.
- Assurance: continuous red teaming, control testing, model validation, third-party attestation such as ISO/IEC 42001 certification, and mapping to SOC 2 or NIST AI RMF evidence.
Worked example
A fine-tuned assistant begins emitting real customer email addresses. Contain first: disable the endpoint or force the safe fallback model, preserve logs, and scope which records leaked. Root cause: fine-tuning on unredacted transcripts. Eradicate by retraining on redacted data with memorization testing; notify per GDPR if personal data was exposed; add a pre-deployment extraction test to the release gate. The post-incident review feeds the assurance file with evidence the gate now exists.
Traps
- Running generic IT incident steps with no model rollback or tool-disable plan.
- Remediating before capturing logs and version state.
- Treating a one-off annual red team as sufficient assurance.
- Missing regulatory notification clocks because the incident was logged as a quality bug.