Incident Command Gap During a Payments Outage
At 02:14 UTC on Saturday, the payments API on IBM Cloud returns 5xx errors for 35% of requests. The incident commander role is not assigned until 02:32, after a customer escalates. Two engineers independently restart pods in the Dallas cluster; one scales a deployment from 12 to 30 replicas, which worsens database connection contention. The status page is updated once at 02:50. Service is restored at 03:06 by rolling back a feature flag. The postmortem draft says the root cause is that an engineer ran the wrong command during a routine change. The timeline has a 14-minute gap and no record of who approved the change. Leadership asks why the same team had a similar database contention incident six weeks earlier.
1. Assess the incident response against severity, roles, and communication expectations, and state what you would change for the next incident.
Read the worked answer
A 35% payment failure rate is a high-severity customer-impacting incident, so command should have been established immediately, not 18 minutes later. The response lacked a single incident channel, a named incident commander, an operations lead, a communications lead, and a scribe. Two engineers acted independently, and scaling to 30 replicas without checking the database pool made the failure worse. The status page update at 02:50 was too late and too infrequent for a payment outage. For the next incident, declare severity from customer impact, assign the incident commander in the first five minutes, and have that person stop uncoordinated changes. Set a communication cadence, for example every 15 minutes for a high-severity incident, and record decisions, approvals, and timestamps in the incident channel. Use a rollback-first runbook when a recent change is suspected.
2. Rewrite the root-cause statement into contributing factors and propose action items with owners and due dates.
Read the worked answer
The draft root cause stops at human error and therefore cannot be fixed. A better statement is that a routine change enabled a code path that exhausted database connections; the deployment had no connection-pool guardrail; the rollback was not automated; and incident command was delayed. Contributing factors also include scaling a stateless tier without checking a shared dependency, and missing approval evidence. Action items should be specific and owned. Examples: add a database connection limit and pool-size check to the deployment pipeline, owner platform team, due in two weeks; add a feature-flag kill switch and test it, owner payments team, due in one week; require incident commander assignment within five minutes for severity 1, owner SRE manager, due in 30 days; automate timeline capture from the incident channel, owner tooling team, due in 45 days. Track completion in the reliability review, not only in the postmortem document.
3. How would you verify that the postmortem and earlier action items actually improved reliability?
Read the worked answer
Verification must go beyond closing tickets. Check whether each action item produced a measurable control: the connection guardrail should appear in pipeline logs and block an oversized pool; the kill switch should be exercised in a game day; incident command should be tested in a simulated severity 1. Track recurrence of database contention, time to assign an incident commander, time to first customer communication, and time to mitigate. Compare those metrics with the prior six weeks. Review action-item completion and evidence in the monthly reliability review, and reopen items that were marked done but have no test or monitoring. If the same class of incident recurs, treat the postmortem as incomplete and redo the contributing-factor analysis rather than adding another untested action item.