Incident Response and Postmortems
What the exam tests
This domain tests whether you can run an incident under pressure and turn it into durable improvement. Expect severity levels, incident roles, mitigation before diagnosis, communications cadence, evidence preservation, blameless postmortems, contributing factors, and action tracking.
Core rules
- Severity is based on user and business impact, not on the suspected cause. Declare early; it is easier to downgrade than to recover lost time.
- Assign an incident commander who coordinates rather than debugs. Add a communications lead, operations lead, and scribe for larger incidents.
- Mitigate first: roll back, fail over, disable a feature, scale out, or shift traffic. Root-cause analysis comes after impact is contained.
- Communicate on a predictable cadence through the status page and stakeholder channels. State impact, current action, and next update time.
- Preserve evidence: deploy IDs, logs, metrics, flow logs, and timeline events before cleanup destroys context.
- Postmortems are blameless and focus on contributing factors, detection gaps, response gaps, and preventive actions with owners and dates.
Worked example. Checkout 5xx rises to 8% after a deploy. The incident commander declares Sev1, communications posts an update, and operations rolls back the canary. Errors fall within six minutes. The postmortem finds the canary gate used a one-hour average error rate, which hid a 1% canary failure. Action: gate on five-minute and one-hour SLO burn, and block promotion when either threshold is exceeded. Reasoning: mitigation precedes analysis, and the postmortem fixes detection and process, not only the code.
Traps that cost marks
Debugging before mitigating; letting the incident commander also be the primary fixer; skipping communications; writing "human error" as the root cause; action items without owners or due dates; holding postmortems only for the highest severity; losing logs through short retention.
Key points
- Severity reflects impact; declare early and assign an incident commander.
- Mitigate first, then investigate root cause.
- Communicate impact and next update time on a fixed cadence.
- Blameless postmortems identify contributing factors and control gaps.
- Every action item needs an owner, priority, and due date.