Harness Resilience Testing Configuration
What this domain tests
This domain covers the concrete Harness setup: scoping a project, installing chaos infrastructure, connecting a ChaosHub, defining an experiment, attaching probes and steady-state checks, setting schedules, and applying governance.
Core rules
- Work in the right scope: account, organization, project, and environment. A common failure is a correctly built experiment pointed at the wrong environment or cluster.
- Create or select chaos infrastructure in the target environment. The delegate or runner needs permissions to create chaos resources, read the target workloads, and clean up.
- Use a ChaosHub or repository for fault and experiment templates; tune the template rather than running defaults. Version experiment YAML in Git when using experiment-as-code.
- Configure target selectors with namespace and labels, fault tunables, duration, and probe definitions. Define abort behavior for failed probes.
- Schedule once, on demand, or by cron; avoid overlapping runs that compound blast radius. Apply RBAC and ChaosGuard policies to restrict who can run what and where.
- Review run history and resilience score; feed findings back into the experiment or service.
Worked example
A payments team wants a daily pod-delete test in the payments namespace. They create a project-scoped environment, attach a delegate running in the payments cluster, and import a pod-delete template from ChaosHub. They set the target to app=payments-api, mode fixed=1, duration 120 seconds, and a steady-state probe that queries Prometheus for error rate. The experiment is scheduled at 02:00 with an abort if the error-rate probe exceeds 1.5%. Reasoning: the infrastructure is in the target cluster, the selector is narrow, the probe is user-facing, and the schedule avoids peak traffic and overlapping runs.
Traps
- Installing chaos infrastructure in a different cluster from the target workloads.
- Missing service-account or RBAC permissions, causing silent probe or cleanup failures.
- Leaving template defaults such as namespace, duration, or target mode unchanged.
- Scheduling a cron experiment without checking for overlapping runs or deployment windows.
- Assuming a probe passed because the experiment completed; check probe status and resilience score.
- Using a broad project scope or missing guardrails that allow production experiments without approval.