Fault Injection and Experiment Design
What this domain tests
This domain focuses on choosing the right fault, parameterizing it realistically, and designing an experiment that isolates cause and effect. The exam tests fault categories, Harness tunables and target modes, control and treatment design, duration, and cleanup.
Core rules
- Map the hypothesis to a fault that exercises the suspected weakness: pod delete for replica loss, container kill for restart behavior, CPU or memory stress for saturation, network latency, loss, or corruption for dependency degradation, DNS error for resolution failure, HTTP abort or delay for application errors, cloud stop or failover for infrastructure loss.
- In Harness, faults are reusable definitions with tunables such as duration, chaos interval, CPU percentage, memory amount, latency, packet loss, and target mode. Target mode can scope to one, all, a fixed count, a fixed percentage, or a random maximum percentage of matching pods.
- Prefer one primary fault per experiment; combine faults only when the hypothesis is about a realistic compound failure and you can still attribute the result.
- Define the observation window before injection, including ramp-up, steady injection, and recovery. Include a control run or baseline period when possible.
- Make cleanup explicit: delete the chaos pod, remove network rules, restore resources, and verify recovery.
Worked example
A recommendation service depends on a catalog API with a 200 ms p95 SLO. The team suspects timeout sensitivity. They inject 150 ms of network latency on 25% of recommendation pods for five minutes, with probes for p95 latency below 600 ms and error rate below 2%. They choose fixed-percent mode rather than all pods, because the hypothesis is about partial dependency degradation. During the run p95 reaches 520 ms and errors stay at 0.8%; the service degrades but remains within tolerance. If they had injected 500 ms latency, the experiment would test a different hypothesis, timeout and circuit-breaker behavior, and should be designed separately.
Traps
- Injecting several faults at once and then guessing which one caused the failure.
- Using an unrealistic duration or severity that tests nothing operationally.
- Targeting all pods by default, which destroys the ability to compare and can exceed the intended blast radius.
- Forgetting that a pod-delete fault may be absorbed by a rolling deployment or PodDisruptionBudget; verify what was actually injected.
- Designing probes that watch the fault injector rather than the service.