Observability
What this domain tests
This domain tests whether you can instrument a chaos experiment so the hypothesis is actually testable. It covers golden signals, metrics, logs, traces, probe queries, baselines, and the difference between monitoring the system and monitoring the fault.
Core rules
- Use user-facing signals: latency, traffic, errors, and saturation. For services, add business KPIs such as order success rate or queue depth when they define normal behavior.
- Capture a baseline before injection, then compare the same signals during and after. Use a control group when the experiment design allows it.
- In Harness, probes can query Prometheus, HTTP endpoints, commands, and Kubernetes state. Write the query and expected threshold before the run; avoid thresholds derived from the same run.
- Correlate metrics with logs and traces to explain a deviation, not just detect it. A latency spike with no error increase may indicate saturation; an error spike with normal CPU may indicate a dependency or DNS fault.
- Keep dashboards focused on the hypothesis and the blast radius. Add run annotations so later analysis can separate chaos effects from unrelated deploys or traffic shifts.
Worked example
An experiment injects DNS errors into a checkout namespace. The team uses a Prometheus probe for HTTP 5xx ratio, an HTTP probe against the checkout health endpoint, and a log query for dial tcp errors. Baseline 5xx is 0.3%. During injection 5xx rises to 4.1% and the HTTP probe fails; logs show DNS resolution failures. The hypothesis fails, so the experiment aborts and the team fixes DNS caching before rerunning. Reasoning: the probes measure user impact and the logs provide causal evidence; the baseline gives a valid comparison.
Traps
- Watching only pod CPU or restart counts, which may rise without user impact or stay flat during a dependency failure.
- Using averages instead of percentiles for latency.
- Setting probe thresholds so loose that every run passes, or so tight that normal variance fails.
- Missing scrape gaps, clock skew, or stale metrics during short faults.
- No annotation of experiment start and end, making dashboards ambiguous.
- Alerting on the chaos injector itself and mistaking that for service impact.
Steady-State Validation
What this domain tests
This domain tests how to define, validate, and interpret the steady-state hypothesis in Harness Chaos Engineering. Expect questions on probe selection, tolerance windows, baseline versus during-chaos comparison, recovery checks, and resilience score interpretation.
Core rules
- The steady-state hypothesis is a set of measurable conditions that define acceptable behavior. Write it before the experiment and make it falsifiable.
- Prefer symptom-based conditions: success rate, p95 latency, queue depth, conversion rate. Avoid cause-based conditions unless they explain a symptom.
- Set tolerances explicitly: a service may allow a transient dip during pod deletion but must recover within a stated window.
- Validate in three phases: baseline before injection, during injection, and recovery after injection. A run that ends while the system is still degraded is not a pass.
- In Harness, probes evaluate the hypothesis; weights and outcomes contribute to the resilience score. A high score is only meaningful relative to the probes you chose and the thresholds you set.
- Abort or roll back when a probe breaches a safety threshold, not merely when the hypothesis fails.
Worked example
An order service defines steady state as success rate above 99.5% and p95 below 400 ms. The experiment deletes one of five pods for two minutes. During the run success rate drops to 99.2% for 30 seconds, then returns to 99.7%; p95 peaks at 380 ms. If the tolerance is no sustained breach below 99.0%, the run passes; if the tolerance is never below 99.5%, it fails. The reasoning is that the threshold must be stated in advance and must distinguish a transient from a sustained breach. The team also checks recovery for five minutes after injection.
Traps
- Writing the steady-state hypothesis after seeing the results, or moving thresholds to force a pass.
- Using too many probes without weighting or interpretation; contradictory probes make the resilience score opaque.
- Checking only during injection and skipping recovery.
- Treating a single failed probe as proof of a design flaw without checking probe reliability.
- Defining steady state from infrastructure metrics that do not reflect user experience.
- Assuming a passing hypothesis means the service is resilient to all faults; it only validates the tested conditions.
Blast Radius Control
What this domain tests
This domain tests how you bound the potential damage of a chaos experiment: target scope, fault severity, duration, environment progression, guardrails, and abort mechanisms.
Core rules
- Blast radius is the set of users, services, and dependencies that could be affected. Control it with narrow selectors such as namespace, labels, and node selectors; target modes such as one, fixed count, or percentage; and short durations.
- Progress from lower-risk to higher-risk: development, staging, production canary, then wider production. Do not start with all pods in production.
- Define abort conditions tied to user-facing signals and make them automatic where possible. Include a manual stop and a cleanup plan.
- Account for shared dependencies: a database, cache, DNS, or message broker can extend blast radius beyond the selected pods.
- Apply governance: RBAC, approvals, ChaosGuard policies, schedules that avoid peak traffic and deployment windows, and communication with responders.
- Increase scope only after a smaller experiment passes and the team understands the failure modes.
Worked example
A team wants to test network latency on the checkout service. They start in staging with one pod for 60 seconds at 100 ms latency. After the steady-state hypothesis passes, they run in production against a canary pod receiving about 5% of traffic for five minutes, with an abort if the 5xx rate exceeds 1%. They do not target all checkout pods because the hypothesis is about canary resilience and the shared payment database is outside the blast radius. Reasoning: each step increases exposure only after evidence, and the abort threshold protects users.
Traps
- Using a label selector that matches more pods than intended, especially after a deployment adds labels.
- Running a long-duration or high-severity fault first because it is faster.
- Forgetting that chaos infrastructure permissions can affect cluster-wide resources.
- No abort or rollback path, or an abort that depends on a human noticing a dashboard.
- Overlapping experiments that multiply the blast radius.
- Assuming namespace isolation prevents dependency impact; it does not.