The Incident Report With a Name in It
A payments provider suffered a four-hour outage after a configuration change to a database connection pool. At 02:14 the alert fired and the on-call engineer was paged; the runbook had no entry for the symptom, so escalation to the database team happened at 03:40 and they joined at 04:10. Service was restored at 06:05. Afterwards the engineering director asked HR to record the on-call engineer's name in the incident record and demanded a written explanation from the person who approved the change. In the following quarter, two near-misses went unreported, on-call shift swaps rose 60%, and mean time to acknowledge alerts went from 9 to 34 minutes. A senior engineer's recommendation is that people simply need to be more careful.
1. Explain, with mechanisms, why mean time to acknowledge alerts rose after the director's response. Connect this to the Second and Third Ways.
Read the worked answer
The Second Way is about fast, rich feedback flowing from right to left; the Third Way is about continual learning, experimentation and psychological safety. The director's response attacks both.
Mechanism: the incident record now carries a name, so the record has become a liability rather than a source of learning. Acknowledging an alert makes you the visible owner of a possible failure, so the rational move is to wait, hope someone else picks it up, or let it escalate. MTTA rising from 9 to 34 minutes is not carelessness; it is a predictable response to increased personal risk.
The same logic explains the other two symptoms. Near-misses go unreported because reporting one means volunteering for scrutiny. Shift swaps rise 60% because on-call has become hazardous.
"Be more careful" completes the pattern: it locates the failure in individual attention rather than in the system that allowed an unvalidated connection-pool change to reach production without staged rollout or a runbook entry. The defences were thin; the people were not the defect.
2. Redesign the post-incident process. What artifacts do you produce, what questions get asked, who attends, and what gets published?
Read the worked answer
Timing and facilitation. A blameless review within 48 hours, run by someone trained to redirect "who" into "what made this possible". Attendees: the on-call engineer, the change author, the database team, a representative from an unrelated team, and a manager who does not own the action items.
Artifact: a timeline built from logs and chat, not memory. Then contributing factors - plural, because a single root cause is almost always a fiction here: no validation of pool size in CI, no staged rollout, no runbook entry for the symptom, escalation path unclear.
Questions. What did we expect to happen? What actually happened? What made the change possible to ship? What made detection take 86 minutes? What made repair take nearly two hours? Which defences would have caught this earlier, and who owns each?
Output. Action items with named owners and dates, tracked in the same backlog as feature work, plus a new runbook entry written from the timeline. Publish the review in a searchable internal library.
Never include individual names as causal factors. The record should make the system smarter, not the engineer quieter.
3. How would you judge whether the new process is working? Give leading and lagging indicators, and say what would count as failure.
Read the worked answer
Leading indicators - these move first and tell you whether behaviour is changing: number of near-misses reported (should rise), number of postmortems published per month, median time from incident to published review, share of action items closed by their due date, on-call shift-swap rate, and MTTA.
Lagging indicators - these tell you whether outcomes improved: MTTR, repeat-incident rate (the same contributing factor appearing twice), change failure rate, and the share of changes with an automated validation or rollback path.
Failure looks like: postmortem volume collapsing to zero, which usually means reporting stopped rather than incidents stopped; action items accumulating without closure; MTTA flat at 34 minutes; and the same connection-pool class of incident recurring. Any one of these says the process is being performed rather than used.
A specific trap: judging the process by incident count alone. A blameless process often reveals more incidents at first because people finally report them. The honest test is whether repeat incidents fall while reporting rises - that combination is the signal that learning is actually happening.