Latency Spike With Two Plausible Causes
At 09:40 the checkout service was deployed with a change to tax calculation. At 09:52, p99 latency rose from 220 ms to 1,800 ms and the error rate went from 0.4% to 2.1%. At 09:55 a third-party DNS provider posted 'degraded resolution' for the region. Rollback takes 12 minutes and puts checkout into read-only mode for 4 minutes; during a previous read-only window, 900 carts were abandoned. The team has 90 minutes before a scheduled peak sales window. Two engineers are arguing: one wants to roll back immediately, the other wants to wait for DNS to recover because 'the timing is suspicious.' No one has yet looked at whether the DNS degradation affects the specific hostnames checkout calls.
1. How do you sequence the next 15 minutes, and what evidence do you require before committing to a rollback?
Read the worked answer
The trap is treating this as a vote between two people. It is a question with a cheap test.
Next 15 minutes, in order:
- Check blast radius first (2 min). Is the latency confined to the tax-calculation path, or is the whole checkout affected? If unaffected services on the same DNS provider are healthy, the DNS explanation is weakened immediately.
- Resolve the hostnames (1 min). The decisive fact is whether checkout's actual outbound calls were failing to resolve during 09:52-09:55. A status-page post about a region is not evidence about your hostnames. If resolution succeeded, the DNS lead is dead and you stop spending attention on it.
- Compare to the last deploy (3 min). Did the previous deploy of the same service produce any latency movement? If tax calculation is on the critical path and the new code adds an external lookup, that is a direct causal story.
- Decide on a tripwire, not a feeling (2 min). Pre-commit to a threshold: e.g. if p99 exceeds 2,000 ms or error rate exceeds 3% before T-30, roll back regardless of cause.
Do not roll back purely on correlation. A 12-minute rollback that induces a 4-minute read-only window during ramp-up is itself a revenue event. But do not wait for certainty either - you will not get it in the window.
The judgement: rollback is the correct default when (a) the change is the only recent variable on the failing path and (b) the cheapest way to test causation is to revert. Here the deploy is on the path and rollback is reversible. So the question is only whether you can afford the 4-minute read-only window now versus risking it during peak. Earlier is cheaper.
2. You must notify stakeholders at T-60. Draft the substance of that notification, and state what you will explicitly not claim.
Read the worked answer
What to send (substance, not wording):
- Impact: p99 latency 1,800 ms versus 220 ms baseline; error rate 2.1% versus 0.4%; checkout is degraded, not down.
- Timeframe: onset 09:52, still ongoing at time of writing.
- What we know: a checkout deploy landed at 09:40 and is the leading suspect; a third-party DNS advisory was issued at 09:55 and is being tested against our actual hostnames.
- What we are doing: the disambiguation test is running now; a decision gate is set at T-30 with a written rollback threshold.
- What stakeholders should do: hold non-essential traffic-driving campaigns until the next update; expect an update at T-30 regardless of outcome.
What not to claim:
- Do not name a root cause. 'DNS issue' and 'bad deploy' are both unproven at T-60, and a wrong cause in writing becomes the story people defend for the next hour.
- Do not promise a recovery time. You have a decision gate, not a fix time.
- Do not describe the incident as resolved-adjacent language ('minor', 'stabilising') while error rate is 5x baseline.
The judgement being tested: a status update is a commitment device. Commit to the next update time and the decision threshold, not to a diagnosis.
3. The incident closes with no confirmed root cause. What is your follow-up conclusion, and how do you avoid a repeat?
Read the worked answer
Accept that 'no confirmed root cause' is a finding, and write it down as one.
Reasoning: two hypotheses stayed live and the system recovered - whether because of the rollback, DNS recovery, or traffic patterns, nobody can now separate. If you declare the deploy guilty, you will fix the wrong thing and skip the DNS dependency question. If you declare DNS guilty, you will skip the code path. Both are guesses dressed as conclusions.
Actions that hold regardless of cause:
- Instrument the ambiguity away. Record DNS resolution failures per hostname as a first-class metric; today you inferred them from a vendor status page. Add deploy markers onto the same latency dashboard so the next disagreement resolves in seconds.
- Make rollback cheaper. The 4-minute read-only window is why this was a debate. Find the stateful write that forces read-only and decouple it; a rollback that costs nothing gets taken without an argument.
- Rehearse the decision, not the tooling. The failure mode here was a 90-minute window spent on advocacy. A pre-agreed tripwire and a named decider would have compressed it.
- Record the confidence level. State plainly in the post-incident review that causation is undetermined, with the evidence for each hypothesis and what would have discriminated between them.
The judgement: an honest 'undetermined, and here is how we would know next time' outperforms a tidy but unsupported root cause, because the tidy version closes the investigation.