Kill a Node on Black Friday
Three weeks before Black Friday, a VP asks the SRE team to "prove resilience" by killing a node in production during peak. The platform spans three AZs in one region. The order service is stateful with a single-primary Postgres and an async replica; measured failover is about 45 seconds. PCI-scoped workloads share the same cluster. There is no production-scale staging environment, no abort tooling beyond kubectl, and the last DR test was 14 months ago. A change freeze runs Nov 20 to Dec 2. The team has three weeks and one engineer who has run fault injection before.
1. What experiments do you actually run in three weeks, and what do you refuse?
Read the worked answer
Refuse: the node kill at peak, as specified. With no abort tooling, co-located PCI workloads, a single-primary database, a freeze in force and no defined steady state, that is not an experiment - it is an unplanned outage with a hypothesis attached afterwards. A chaos experiment is only valid if you can state the steady state, stop the fault, and observe the effect.
Run instead:
- Production-shaped, non-customer environment first - a canary cell or an isolated namespace with the same topology. If no production-scale environment exists, that gap is the headline finding to report, not something to work around silently.
- The failure you already suspect will fail: a scheduled replica-promotion drill. Measure real promotion time against the 45-second claim and convert it to budget. At 99.9% over 28 days the budget is ~40 minutes, so 45 seconds is affordable; at 99.95% (~21 minutes) repeated failovers start to matter.
- Pod/process kills and network latency injection into a non-critical dependency, then into a single canary instance of the order service at low traffic.
- AZ-level fault only after 1–3 pass, on the canary cell, in the lowest-traffic window, with a written freeze exception.
The experiment worth running produces a number you will act on: does failover hold the SLO, and what is the customer-visible effect of losing 5% of traffic?
2. What preconditions and guardrails must exist before any production-shaped experiment?
Read the worked answer
- Hypothesis and steady state written down in SLIs - error rate, p99, throughput - with a predicted outcome. "The system should survive" is not a hypothesis.
- Blast-radius cap: a hard traffic percentage (for example ≤5% of orders, one canary cell) and a hard duration (for example 5 minutes) with automatic revert.
- Abort conditions plus one person with authority to stop. Automatic abort on burn-rate or error-rate thresholds; a named human, not a committee.
- Live observability during the run: a dashboard showing the injected fault, the SLIs and the abort thresholds, watched by someone who is not operating the injector.
- Comms and approvals: support and status page pre-briefed, PCI/security sign-off on which workloads are in scope, change approval filed, rollback rehearsed, and a documented freeze exception if applicable.
- A programme stop rule: if the first experiment shows failover does not meet the SLO, fix that before attempting anything more aggressive. Otherwise you are measuring a broken system for sport.
3. How do you respond to the VP in the room?
Read the worked answer
Reframe in their language: the goal is knowing whether we survive Black Friday; the cheap way to learn that is a drill that can be stopped, and the expensive way is learning it live with customers. Both paths answer the question - only one of them risks the revenue.
Offer a concrete counter-proposal: "On , in a canary cell carrying 5% of orders, with automatic abort on error rate, we kill a node and measure whether Postgres failover holds the SLO. You get the number the same day. Based on it we either schedule a peak-traffic experiment with proper abort tooling for the next event, or we go fix failover first."
Be precise that the peak version is currently untestable rather than refused on principle, and name the blockers, because they are fixable: failover path unverified, no automatic abort, PCI workloads co-located. If leadership still insists, ask for the risk acceptance in writing against a named owner - not as passive aggression, but because that is how the tradeoff gets recorded and revisited when the next request arrives.