The Ten Percent That Hit Production
A shared Kubernetes cluster has 120 nodes and 8 tenants. The orders service has 80 pods: 60 in production and 20 in staging. A Harness chaos experiment uses the label selector app=orders, a 10% pod percentage, and no namespace or environment selector. It selects 8 pods, kills them for 5 minutes, and restarts them. Because production and staging share the app=orders label, 6 of the killed pods are production and 2 are staging. The intended target was staging only. During the run, production orders p99 rises from 200 ms to 1.2 s, and two tenants report dropped checkouts. The cluster has separate node pools for staging and production, but the experiment did not use a node selector. The team wants to add a 10% cap and rerun.
1. What specifically allowed the blast radius to exceed intent, and which controls would have prevented it?
Read the worked answer
The selector matched every pod with app=orders across both environments. The 10% was applied to the union of 80 pods, so 8 pods were selected, and 6 of them were production. There was no namespace selector, no env=staging label, and no node selector, so nothing constrained the target to staging. Adding only another 10% cap would not fix the root cause because 10% of the same union is still 8 pods.
Controls that would have prevented it: a namespace selector for orders-staging, an env=staging label, a tenant label, a node selector for the staging node pool, a fixed target of 2 pods (10% of 20), and a Chaos Guard policy denying any experiment where env=prod or namespace is not staging. A pre-flight target count would also have exposed the 6 production matches before execution.
2. Redesign the experiment target and guardrails, then describe a pre-run verification.
Read the worked answer
Use selectors namespace=orders-staging, env=staging, app=orders, and the specific tenant label. Set the target to 2 pods, or 10% of the 20 staging pods, and add a node selector for the staging node pool. Keep the pod-delete fault at 5 minutes, and define abort conditions on staging SLOs plus a cross-check that production p99 and error rate remain at baseline.
Guardrails: Chaos Guard should deny execution if env=prod, if the namespace is not a staging namespace, or if the computed target exceeds 2 pods. Pre-run verification: run a read-only label query in Harness or kubectl to count matching pods and confirm exactly 2 matches; verify none are in production; verify the node pool; then run a 1-pod canary before the full 2-pod experiment. Do not rely on the percentage alone; verify the actual selected set.