Harness Certified Chaos Engineering cheat sheet

Chaos engineering is the discipline of experimenting on a system to build confidence in its ability to withstand turbulent conditions in production.

The steady-state hypothesis defines normal behavior via measurable business or system metrics; experiments test whether the hypothesis holds under fault.

Blast radius is the scope of impact an experiment can cause; start small (one pod, one node) and expand only after validating safety controls.

The experiment loop: define steady state, inject fault, observe deviation, learn, and either fix weaknesses or adjust the hypothesis.

Chaos engineering differs from testing: tests assert known outcomes; chaos experiments explore unknown failure modes and validate resilience.

Principles: build a hypothesis, vary real-world events, run experiments in production (or production-like), automate experiments, minimize blast radius.

Common fault types: resource stress (CPU, memory, disk, network), state faults (pod kill, container kill), and network faults (latency, packet loss, corruption).

Game days are planned exercises combining multiple faults; they validate incident response, observability, and runbooks alongside system resilience.

Prerequisites: strong observability, automated rollback, defined SLOs, and a culture that treats experiment failures as learning opportunities.

A well-formed experiment specifies: hypothesis, steady-state metrics, fault type, target scope, duration, and abort conditions.

Fault injection should be time-boxed and reversible; always define a rollback or halt mechanism before starting.

Network faults: latency adds delay; packet loss drops packets; corruption alters packet content; bandwidth limits throughput; DNS faults break resolution.

Resource faults: CPU stress consumes cores; memory stress allocates RAM; disk stress fills or slows storage; IO stress delays read/write operations.

State faults: pod-kill deletes a pod; container-kill restarts a container; node-drain evicts workloads; these test restart and rescheduling behavior.

Experiment design must include a control group or baseline period to distinguish fault impact from normal variance.

Use gradual ramp-up of fault intensity (e.g., 10%, 50%, 100%) to observe thresholds and avoid abrupt system collapse.

Hypothesis format: 'Given [conditions], when [fault] occurs, [metric] will remain within [threshold] for [duration].'

Avoid combining multiple independent faults in a single experiment unless testing compound failure scenarios explicitly.

Harness Chaos Engineering (HCE) uses ChaosHub to store and share experiment templates and fault definitions.

Experiments are defined as YAML manifests with a ChaosEngine resource that references a ChaosExperiment and target selectors.

The ChaosEngine spec includes: experiment name, engineState (active/stop), appinfo, targets, and chaosServiceAccount.

Target selectors use labels, namespaces, and kind to scope which pods or nodes receive the fault.

Probes in Harness chaos validate steady state before, during, and after injection; probe types include HTTP, command, Prometheus, and Kubernetes.

Resilience probes can be configured to halt the experiment if a metric breaches a threshold, enforcing automatic blast-radius control.

Harness integrates with CI/CD pipelines to run chaos experiments as pipeline stages, gating deployments on resilience validation.

Experiment scheduling supports cron-based recurring runs; use this to continuously validate resilience rather than one-off tests.

Templates in ChaosHub can be customized per environment; always pin template versions to avoid unintended changes during experiments.

Observability for chaos requires metrics, logs, and traces correlated by experiment ID and target labels.

Key metrics to monitor: error rate, latency (p50/p95/p99), throughput, saturation, and resource utilization.

Distributed tracing reveals how faults propagate across service boundaries; use trace sampling to capture fault-affected requests.

Harness integrates with monitoring tools (Prometheus, Datadog, New Relic, Grafana) to display experiment impact in real time.

Probes should query the same observability stack used for steady-state validation to ensure consistency.

Log correlation: inject experiment metadata into logs so fault-induced errors can be filtered and analyzed.

Alerting must be configured to distinguish experiment-induced anomalies from real incidents; use maintenance windows or label-based suppression.

Observability gaps are a common cause of false steady-state validation; verify coverage before running experiments.

Steady state is defined by a set of metrics and their acceptable ranges during normal operation, not by the absence of faults.

Validation must occur before injection (baseline), during injection (continuous), and after injection (recovery).

Probes are the mechanism for steady-state validation; they run assertions against metrics, endpoints, or commands.

A probe failure can be configured to halt the experiment, mark it failed, or continue with warning, depending on risk tolerance.

Steady-state metrics should be business-relevant (e.g., order success rate) and system-relevant (e.g., error rate, latency).

Avoid using a single metric; use a composite hypothesis covering availability, latency, and correctness.

Recovery validation confirms the system returns to baseline within an expected time; lingering degradation indicates a resilience gap.

Baseline periods should be long enough to capture normal variance (e.g., diurnal patterns) and avoid false positives.

Blast radius is limited by target selectors: namespace, label, pod count, node count, and percentage of replicas.

Start with a single pod or a canary instance; expand only after confirming no unexpected impact.

Use experiment duration limits and automatic stop conditions (probes) to prevent runaway faults.

Harness supports blast radius constraints via target mode (one, all, fixed, percent) in the ChaosEngine spec.

Isolate experiments to non-production environments first; promote to production only with mature observability and rollback.

Dependency mapping: identify upstream and downstream services that could be affected by the target; include them in validation.

Kill switch: ensure a manual or automated stop is available and tested; the ChaosEngine engineState can be set to stop.

Blast radius control also includes time-based limits (experiment timeout) and resource quotas to prevent cluster-wide impact.