The Same Outage, Three Times
In six months, a payments service has had three near-identical incidents: a configuration push to production flipped a routing flag, 30-40% of transactions failed for 12-45 minutes, and rollback took longer than the change itself. Each incident produced a postmortem. The action items were "engineers should double-check config changes," "add a second reviewer," and "remind the team to follow the process." All three are marked closed; two were closed with no code or config change, only a Slack message. Blame is genuinely low - the same two engineers were involved each time - but so is learning. The next config push is Thursday.
1. Evaluate the three action items. Why did they fail, and what would a real action item look like?
Read the worked answer
All three are changes to human behavior, and none is falsifiable.
- "Double-check config changes" - there is no way to verify this happened, no way to know it worked, and no mechanism. Given a 45-minute outage and a Friday afternoon, every engineer will double-check. Given a normal Tuesday, nobody will.
- "Add a second reviewer" - a reviewer who is looking at a diff cannot see that
routing.flag=truewill send 40% of traffic into a black hole. Review catches typos and intent mismatches, not semantics. This action item looks like a control and functions as a ritual. - "Remind the team to follow the process" - the process was followed. That is what makes this one damning: it asserts the failure was a discipline problem when the evidence says it was a system problem.
Why they failed: they were written to close a ticket, not to change a system. Marking them closed after a Slack message is the completion criterion working exactly as designed - the design is wrong.
A real action item is a change to a system, with an owner, a date, and a test that can fail:
- "Config schema validation in CI rejects any routing flag change unless a matching routing test passes. Owner: A. Due: 14 days. Verified by: the next Thursday push cannot merge without it."
- "Canary push to 1% of traffic with automatic rollback when the error-rate SLI exceeds threshold for 60 seconds. Owner: B. Due: 30 days."
- "Rollback completes in under 2 minutes, measured by a quarterly game day. Owner: C. Due: 60 days."
Note that the third one addresses the actual worst part of the incident: not that the change was wrong, but that it took longer to undo than to make.
2. The culture is blameless but nothing changes. What is missing, and how do you add it without reintroducing blame?
Read the worked answer
Blamelessness is a precondition for learning, not a substitute for it. It removes the incentive to hide information. It does not, by itself, produce a single change to a system. Teams routinely achieve perfect blamelessness and zero improvement.
What is missing is accountability - which is different from blame.
| Blame | Accountability |
|---|---|
| Who caused this? | Who owns the fix? |
| Backward-looking, personal | Forward-looking, structural |
| Ends the inquiry | Extends the inquiry |
Concrete mechanisms to add:
- Named owner and due date on every action item, tracked in the same backlog and the same sprint reviews as feature work. An action item in a separate doc is an action item that will not happen. Feature work always wins against a document.
- A completion standard: "no system change, not done." A Slack message does not close an item.
- A recurring review - 30 minutes monthly, walking open actions and their aging, attended by an engineering manager with authority to reprioritize.
- A repeat-incident signal: when an incident matches the class of a previous one, that reopens the predecessor and escalates it. Track the repeat rate as a first-class metric.
The framing that keeps this blameless: "We are not asking who did this. We are asking why the system made this the easy thing to do, and what we are changing so it isn't." The two engineers involved should be the ones designing the guardrail - they have the most context, and giving them the fix is the opposite of blame.
3. Thursday's push is in two days. What do you do now, and what do you measure over the next 90 days?
Read the worked answer
Now (48 hours):
- Get the change classified before it ships. Someone reads the diff specifically for blast radius: does it touch routing, auth, or money? If yes, it is not a normal push.
- Manual canary. Push to one instance or 1% of traffic, watch the transaction success rate for 10 minutes, then proceed. This is imperfect and slow and it costs an hour. It is also the only guardrail that can exist by Thursday.
- Name a rollback owner and pre-compute the rollback path before the change starts. Write down the expected rollback duration. If it is longer than the push, that is the real finding.
- Do not interpret "be careful" as a mitigation. Ship the manual process and log it as a known gap with an owner.
90-day metrics - pick few, and ones you can actually move:
| Metric | Target | Why |
|---|---|---|
| Config-caused incident count | 0 | The direct outcome |
| Time from change to detection | < 3 min | Detection is the binding constraint today |
| Rollback duration (measured, not estimated) | < 2 min | Fixes the "longer to undo than to make" problem |
| % of postmortem actions closed with a system change | > 80% | Tests the mechanism, not the sentiment |
| Repeat-incident rate | 0 across quarters | The single number that says whether learning happened |
The last one is the honest scorecard. Three identical incidents in six months is not a run of bad luck; it is a system with a known, unaddressed defect. The metric that matters is whether a fourth one ever occurs.