DevOps Foundation study materials
DevOps Foundation study resources

The Blameless Postmortem That Wasn't

DevOps Foundation · Put your knowledge to work in context.

All case studies

The Blameless Postmortem That Wasn't

A 400-person SaaS company runs a payments service. A deploy by a platform engineer triggers a 47-minute outage that blocks checkout and costs an estimated $180K in lost revenue. The engineering director asks the engineer who ran the deploy to write the postmortem alone, ready by the next morning. The published root cause reads 'human error - the engineer skipped checklist step 4'. Within a month, tech leads have added a second manual reviewer to every deploy, deployment frequency has fallen from twelve a week to four, and two engineers have privately said they will stop volunteering for release duty. The director asks you why velocity dropped and wants the reviewers removed.

1. Diagnose the drop in deployment frequency after the postmortem. What is the causal chain, and where in it would you intervene?

Read the worked answer

The chain

  1. The postmortem named a person as the cause, so the lesson the organisation actually learned was 'deploying is dangerous and the person who deploys is exposed'.
  2. Leads responded rationally to that lesson with local risk reduction: a second manual reviewer on every deploy.
  3. Manual gating lengthened the path to production, so deploys batched up and frequency fell from twelve to four a week - which increases the risk per deploy, the opposite of the intent.
  4. Engineers are now avoiding release duty, which concentrates the risk on whoever remains.

Where 'human error' stops too early

  • Why was checklist step 4 the only thing standing between a mistake and a checkout outage? Where were the automated tests, the canary, the fast rollback?
  • Why is a checklist step skippable under time pressure at all?
  • Why did no alert fire before customers noticed?

Intervention point Fix the system so a skipped step cannot reach production alone, and fix the story so that reporting a mistake is safe. Removing the reviewers by decree while the fear remains will not work - see the next question.

2. Redesign the postmortem process so this does not recur. What are the concrete mechanics?

Read the worked answer

Mechanics

  • Facilitated, not solo. The person closest to the change contributes, but does not own the document, and never authors it alone under a deadline.
  • Multiple perspectives. Include someone from outside the team and someone from support or SRE. The facilitator's job is to keep asking 'how did our defences fail?' rather than 'who did this?'.
  • Timeline first, judgement later. Reconstruct what happened from logs, chat and deploy records before anyone interprets. Publish the raw timeline.
  • Contributing factors, not a root cause. Plural, system-level. 'Single-person deploy with no automated verification and no rollback path' is a finding; 'skipped step 4' is a symptom.
  • Counterfactual reasoning. Ask what would have had to be true for this to have been caught, and how long it would have taken to detect without the customer reporting it.
  • Actions with owners and dates, tracked to completion and reviewed for whether they actually reduced risk. Most postmortems die here.
  • Share across teams. The learning is a company asset; a payments outage teaches every team with a deploy path. Reuse the story in onboarding and in blameless-review training.

Language matters: name roles and systems, never individuals, and avoid 'should have' phrasing, which converts analysis back into blame.

3. The director wants the second reviewers removed by edict next week. Advise on the approach and on what would actually restore speed.

Read the worked answer

Removing the reviewer by edict treats a symptom and leaves the cause - the belief that deploying is personally risky - in place. Expect the gatekeeping to reappear in another form, for example release freezes or informal 'don't deploy on Friday' norms.

What to do

  1. Ask the leads what they were protecting against. They may have identified a real gap; do not assume the reviewer is pure superstition.
  2. Meet the underlying need with automation rather than people: mandatory pipeline checks, canary with automatic abort, one-command rollback, deploy observability. Then a reviewer adds little and can be retired on evidence.
  3. Change what leaders say and do. If the director's first response to the outage had been 'what in our system let this happen?', the reviewer would never have been added. Leaders describing their own mistakes is the cheapest culture intervention available.
  4. Measure and publish: deployment frequency back up, change failure rate and restore time holding or improving. Show the leads the safety data rather than asserting it.

Distinguish verification from trust. Automated verification of a change is engineering; a second human signing off on a colleague is a substitute for it, and it costs flow without buying much safety.