Practical Chaos Engineering
Chaos engineering is the discipline of experimenting on a system to build confidence in its ability to withstand turbulent conditions. The method is scientific, not destructive: state a hypothesis ("if one instance of the checkout service is terminated, conversion SLI stays within SLO"), define a steady-state metric that represents normal behavior, introduce a controlled variable, and compare against the baseline. Prerequisites are non-negotiable: working observability, an agreed blast radius, an automatic abort condition, a rollback path, and on-call awareness so real incidents are not confused with experiments.
Common fault classes are resource (CPU, memory, disk), network (latency injection, packet loss, partition, DNS failure), process and instance termination, dependency failure (upstream errors, timeouts), zone or region failure, and clock skew. Maturity typically progresses from staging and pre-production, to non-critical production paths, to continuous automated experiments (often as part of delivery pipelines) with a limited blast radius. Game days combine scenario rehearsal with people: they test incident response, communications, and runbooks alongside the systems. Tools candidates should recognize include Chaos Toolkit, LitmusChaos, Gremlin, AWS Fault Injection Service, and Chaos Mesh; the tool choice matters far less than the hypothesis and the abort criteria.
Traps: running experiments with no hypothesis (that is random breakage, not chaos engineering); starting in production at peak traffic; no defined abort condition or steady-state metric; confusing load testing with chaos engineering (load tests measure capacity, chaos tests measure resilience to failure); and "compliance theatre" - a single annual experiment that verifies nothing about real behavior.
Rule of thumb: start with the smallest interesting failure - a single instance, in a staging environment, at low traffic, with the experiment time-boxed and a kill switch - and only widen scope once you can prove the abort actually works.