Partial Mitigation vs. Full Rollback Under a Traffic Deadline
The checkout service began returning errors at 14:05, twenty minutes after a scheduled deploy. Error rate moved from 0.4% to 3.1% and is climbing roughly 0.2 points per ten minutes. Peak traffic starts at 17:00 and is expected at 2.8x current volume. Two remediation paths exist. A feature flag can disable the new pricing path in about two minutes, but only 62% of sessions route through it; the other 38% use the legacy path the same deploy touched. A full rollback is 45 minutes of work and needs a 20-minute maintenance window, because the release included a schema migration that must be reversed. The on-call engineer wants to roll back immediately. The product owner wants to wait for a 16:00 lull and protect a quarterly revenue commitment.
1. What is the correct immediate action at 14:30, and what makes it correct rather than the alternative?
Read the worked answer
Act now with the reversible, low-blast-radius control; do not wait for the lull.
- Pull the flag at 14:30. It costs two minutes and no downtime, and it removes the failing path from 62% of sessions. Speed of harm reduction dominates completeness of fix.
- Instrument immediately after: compare error rate on flagged vs. unflagged cohorts. If the rate on the unflagged 38% stays flat, the flag is a sufficient fix and the rollback becomes a scheduled change, not an emergency.
- If the unflagged cohort is also degrading, the flag was a probe, not a fix, and you roll back at once - the 45-minute clock has only lost four minutes.
The product owner's "wait for the lull" is rejected because the lull is a guess about a system whose error rate is still climbing, and because peak volume is 2.8x - rolling back at 16:30 puts the maintenance window inside the worst possible traffic. The decision rule is: prefer actions that are fast and reversible, and prefer them before actions that are slow or destructive. Flag first, rollback conditionally.
2. At 14:45 the flag is off and the total error rate sits at 2.6% - down from 3.1% but not back to 0.4%. How do you decide whether that is good enough?
Read the worked answer
Decompose before you judge. 2.6% overall is consistent with two very different worlds:
- World A: flagged cohort near 0%, unflagged cohort still at ~3%. The flag worked; the residual is a separate defect on the legacy path.
- World B: both cohorts near 2.6%. The flag did nothing measurable and the earlier "improvement" was noise or the 0.2/10min climb pausing.
Split the metric by cohort and split by endpoint. Then compare against the pre-deploy baseline of 0.4%, not against 3.1% - the baseline is the only number that says whether customers are actually served.
If World B, roll back. If World A, you have a live decision about the legacy path with peak three hours out: rolling back now still fits before 17:00 and is the safer option, because a 3% error rate on legacy checkout at 2.8x volume is a materially worse outcome than a 20-minute maintenance window taken early. The trap here is accepting "it went down" as "it's fixed."
3. What should be communicated, to whom, and in what order over the next two hours?
Read the worked answer
Sequence by who can act on the information, not by seniority.
- Incident channel / support and on-call: current state, what was changed, what the next checkpoint is and when. Give a time, not a promise.
- Product owner: the flag outcome and the evidence for the rollback decision, framed as cost. If you rolled back, the number they care about is 20 minutes of downtime at a stated traffic level, versus continued errors at peak. Give them that arithmetic; do not give them adjectives.
- Customer-facing / status page if the error rate has been user-visible for more than roughly 15 minutes at meaningful volume - customers notice before your executives do.
- Executive sponsor: one page after the technical picture is stable, not during.
Two content rules. State facts and timestamps; never state a root cause you have not confirmed. And separate "what we did" from "what we still don't know," because the second list is what drives the follow-up. Miscommunication in incidents is almost always a missing checkpoint time or a premature causal claim, not a missing status update.