The Green Dashboard That Wasn't
During a staged chaos review, a payments team runs a pod-delete fault against checkout-api in a pre-production cluster. The service has 12 replicas, a 99.95% availability SLO, and a p99 latency target of 250 ms. Before the experiment, the team records a 30-minute baseline: ingress availability 99.97%, p99 180 ms, and 0.02% 5xx responses. The experiment deletes 2 pods every 5 minutes for 20 minutes. The ingress dashboard stays at 99.94% availability and shows no p99 breach, so the run is marked passed. The next day, mobile support reports a 4x increase in checkout 503s during the same window. Mobile traffic goes through a gateway that samples only 5% of requests and is not included in the ingress SLI. One engineer argues that because the SLO was met, the experiment is valid. The team wants to rerun immediately to confirm.
1. Was the experiment's conclusion valid? What must the team establish before rerunning?
Read the worked answer
No. The run answered a narrower question than the team thinks. The only steady-state signal was the ingress SLI, and it excludes the mobile gateway, so 99.94% ingress availability is not evidence that mobile checkout stayed healthy. The support signal (4x 503s) is a hypothesis violation unless the team can show it was unrelated, pre-existing, or statistically normal for that window.
Before rerunning: (1) reconstruct mobile 5xx rate and p99 from gateway logs for baseline and experiment windows; (2) check whether 5% sampling is representative and whether the 4x claim survives confidence intervals; (3) add a gateway probe and a per-client SLO; (4) define pass/fail as 'steady-state hypothesis not violated', not 'SLO still met'. Rerunning first would just generate another ambiguous result and consume error budget.
2. Write a steady-state hypothesis for the rerun with at least three signals and explicit abort conditions.
Read the worked answer
Example hypothesis: for 20 minutes after the first pod delete, checkout availability measured at both ingress and mobile gateway stays >=99.95%, mobile 503 rate stays within 0.02 percentage points of the pre-experiment baseline, and p99 latency at both measurement points stays <=250 ms. A separate queue-depth signal should stay below 500 pending requests.
Abort conditions: 5xx rate >0.5% for 2 consecutive minutes, p99 >500 ms for 3 minutes, or gateway 503 rate >3x baseline for 1 minute. Use a pre-experiment baseline of at least 30 minutes and compare with confidence intervals, not a single point. The abort should stop the fault and mark the experiment failed, not just alert. This matters because the previous run had no mobile signal and no user-facing abort threshold.