Conflicting Signals During a Live Checkout Incident
At 02:14 UTC the on-call engineer for Meridian Pay's checkout API is paged: p99 latency is 1,850 ms against a 240 ms baseline and has been climbing for nine minutes. Two monitoring sources disagree. The APM tool reports a 0.4% error rate; edge proxy logs show 6.1% 5xx responses over the same window. A config-only change shipped at 01:50 altered feature-flag defaults, restarted 3 of 12 pods, and reached a 5% canary. Rolling back takes roughly six minutes, but the rollback also reverts a pricing default that a partner integration started consuming this week. Support has logged 14 tickets in 20 minutes. No second responder is available until 03:00 UTC, and the on-call cannot page the partner's engineering team directly.
1. Decide whether to roll back now or keep investigating. State the decision rule you are applying, not just the outcome.
Read the worked answer
Decision rule: act when the cost of being wrong is bounded and reversible, and the cost of waiting is unbounded and accelerating.
Applying it here:
- Waiting is the unbounded side. Latency has climbed for nine minutes with no plateau, a canary is still ramping, and support volume is rising. If the fault is in the config change, the 5% canary can become 100% on the next ramp with no human in the loop.
- Rolling back is bounded. Six minutes to execute, and the one genuinely expensive side effect - reverting the partner pricing default - is recoverable by re-pushing that single flag after the incident, roughly 15 minutes of work.
- The symptom timing fits the suspect. Ship at 01:50, onset around 02:05, 3 of 12 pods restarted: consistent with a partial-rollout fault.
Action: perform a targeted rollback of the config change on the three restarted pods first, preserving the partner pricing default if the flag system allows per-key revert. If it does not, roll back everything and re-push pricing immediately.
Falsification test to run in parallel: does latency track only the restarted pods? If healthy pods show identical latency, the config change is not the cause and the rollback decision should be revisited within ten minutes.
What would change the answer: evidence the latency began before 01:50, or a hard dependency where reverting the pricing default breaks live partner transactions.
2. The two monitoring sources disagree by a factor of fifteen. How do you decide which number to communicate, and what do you check first?
Read the worked answer
Do not average them and do not pick the more comfortable one.
Why they diverge: they almost certainly measure different denominators. APM instruments application handlers, so requests that die at the edge - upstream timeouts, connection resets, 502s from a saturated proxy - never register as application errors. A load-related latency event produces exactly that shape: edge-level failures with a near-clean application error rate.
Checks, in order:
- Pull 20 edge log lines with 5xx and try to map each to an APM trace ID. If most have no trace, APM is structurally blind to them.
- Confirm whether APM counts a slow 200 as success. If so, the error metric is not measuring what stakeholders assume.
- Compare both against completed checkout volume per minute - the business-truth metric that neither tool owns.
Communication rule: until reconciled, report the higher number and label it as unconfirmed. Under-reporting an incident to stakeholders is the error that damages trust; over-reporting a resolved one is merely embarrassing.
What would change the answer: if edge 5xx map cleanly to trace IDs at a 0.4% rate, the 6.1% figure is a logging artifact - for example retries counted as distinct failures - and the lower number stands.
3. Write the first stakeholder update (under 150 words) and justify each element you chose to include or omit.
Read the worked answer
Model update:
02:30 UTC - Checkout degradation, SEV-2, investigating. Impact: elevated latency (p99 ~1.9s vs 240ms baseline) and failed checkouts. Our two error measurements disagree (0.4% vs 6.1%); the working assumption is the higher figure. Roughly 5% of traffic is on the suspect configuration. Action taken: rolling back the 01:50 config change on the affected pods. One known side effect - a partner pricing default will revert and be re-applied after stabilization. Unknown: whether the fault is configuration or load-driven. Owner: on-call engineer. Next update 03:00 UTC, or sooner if the rollback resolves it.
Justification:
- Timestamp and severity first, because readers triage by those two fields.
- Quantified impact, not adjectives - "elevated latency" alone invites worse guesses.
- Both conflicting numbers, with the conservative one flagged, so nobody later claims they were misled.
- The pricing side effect disclosed pre-emptively; it is the detail most likely to cause a surprise downstream.
- Explicit unknowns and a named owner with a next-update time, which converts an anxious audience into a waiting one.
- Omitted: root-cause speculation and the internal pod topology. Both are noise at this stage and become liabilities if wrong.