Harness Certified Chaos Engineering common mistakes
Harness Certified Chaos Engineering · Understand the traps and how to avoid them.
Treating the Harness chaos experiment as a one-off script run from the UI rather than defining an explicit hypothesis with steady-state metrics, a defined blast radius, and an abort condition before execution.
Candidates often come from a scripting background where a chaos test is just running a fault and watching dashboards; they skip the written hypothesis because the platform makes launching an experiment easy, and the exam then probes whether they can articulate the scientific method behind it.
What to do instead
Before any experiment, write the hypothesis as a falsifiable statement: 'When we inject X fault into Y service for Z duration, the steady-state metric M stays within threshold T.' Attach the metric and threshold to the experiment configuration, define the blast radius explicitly, and configure an automatic abort. Practice converting each experiment you run into this template.
Confusing the fault types Harness ChaosFaults support and their semantics - e.g., treating network latency as packet loss, pod-delete as pod-kill, or assuming a stress fault targets memory when it targets CPU - and selecting the wrong fault for the hypothesis.
The fault library is broad and several faults have similar names but different mechanisms and different observability signatures. Candidates memorize names superficially and pick the one that sounds right rather than mapping the failure they want to reproduce to the correct fault primitive.
What to do instead
Build a personal mapping table: for each fault (pod-delete, pod-kill, container-kill, network-loss, network-latency, network-corruption, DNS error, CPU/memory stress, HTTP chaos, JVM chaos), record the mechanism, the expected symptom in metrics/logs/traces, and the recovery behaviour. Verify each against the platform's fault documentation and test in a lab namespace.
Setting the blast radius to the whole cluster or all replicas of a deployment because scoping to a label selector or a percentage felt like extra work, then being unable to reason about the answer when the exam asks how to limit impact.
Default or convenience configurations often target broadly, and candidates under time pressure skip the selector. They also underestimate how a chaos experiment on shared infrastructure can cascade through dependencies, so they don't see the need to scope.
What to do instead
Always scope experiments with a namespace plus label selector, and where supported use percentage or count-based targeting (e.g., 1 of N pods) rather than all. Explicitly configure and state the blast radius in the experiment definition, and rehearse the reasoning: what is the smallest population that still validates the hypothesis?
Running an experiment with no steady-state validation defined, then judging success by 'the app still looked up' rather than by comparing pre-during-post metrics against a declared threshold.
Candidates conflate 'no visible outage' with 'steady state held'. Without a baseline and a threshold, there is nothing to compare against, and subtle degradation (elevated latency, error-rate creep, retry storms) goes unnoticed.
What to do instead
Define steady-state indicators (latency p99, error rate, throughput, saturation) with numeric thresholds and a baseline window before injection. Configure the experiment to evaluate these during and after the fault, and treat a threshold breach as a failed hypothesis requiring investigation, not a pass.
Failing to configure or verify the abort/rollback condition, so a fault that causes real degradation keeps running until its duration expires.
Candidates focus on the injection and the observation but treat cleanup as automatic. They assume the platform always reverts the fault, and they don't think about what happens when the abort condition itself is misconfigured or the probe can't reach the target.
What to do instead
For every experiment, set an explicit abort condition tied to a steady-state metric, verify the probe/health-check path works before injection, and confirm the revert/cleanup step (fault removal, pod restoration) is defined and tested. Know the difference between duration-based stop and condition-based abort.
Ignoring the observability prerequisites - no dashboards, no baseline metrics, no log/trace correlation - and then trying to interpret experiment results from memory or a single graph.
Candidates treat observability as a separate topic from chaos engineering and skip the setup step, or they assume the platform's built-in reporting is sufficient without wiring the application's own metrics and traces.
What to do instead
Before running experiments, confirm the target's metrics are scraped, dashboards show the steady-state indicators, and logs/traces are correlated by request or trace ID. During the experiment, read results across metrics, logs, and traces together, and be able to explain what each signal would show for a given fault.
Running chaos experiments against production or a shared environment without a rollback plan, approval, or a clear separation between the experiment and the environment's normal change activity.
Candidates want realistic conditions and jump to production, or they run in a shared staging environment without coordinating, then cannot distinguish experiment-induced effects from unrelated changes. They also underestimate the organizational controls the exam expects around production chaos.
What to do instead
Start in a non-production environment with a defined promotion path. For production, require explicit approval, a documented rollback, a narrow blast radius, a defined time window, and a way to distinguish experiment effects from concurrent changes (e.g., change freeze or annotated time window).
Misreading the Harness experiment YAML/structure - mixing up the experiment definition, the fault templates, and the runtime inputs - and then answering configuration questions based on the wrong layer.
The platform separates reusable fault templates from experiment definitions and from runtime parameters. Candidates who have only clicked through the UI don't internalize which field belongs to which layer, so they misplace selectors, durations, or probe definitions.
What to do instead
Study the structure explicitly: identify where the target selector lives, where fault parameters live, where probes/steady-state checks are declared, and where runtime inputs are supplied. Rebuild a working experiment from scratch in YAML and annotate each block's purpose.
Treating a single successful experiment as proof of resilience, without varying fault magnitude, duration, timing, or combining faults to test dependency and cascade behaviour.
Candidates stop when the first run passes, because the hypothesis was validated once. They don't systematically explore the parameter space or test interactions between services, so they miss the failure modes that only appear under combined or prolonged stress.
What to do instead
Design a progression: start with a small fault, increase magnitude/duration, then test combinations (e.g., one dependency degraded while another is unavailable). Record each run's parameters and outcome so you can reason about thresholds and cascades rather than a single data point.
Not distinguishing between a fault that is correctly injected but not observed (instrumentation gap) and a fault that was never injected (configuration or targeting error), and answering as if the application simply resisted the fault.
Candidates see no impact and conclude the system is resilient, when the real cause is a selector that matched no pods, a namespace mismatch, a missing sidecar, or metrics that weren't being collected. The exam tests whether they can diagnose the experiment itself.
What to do instead
After every run, verify the fault actually reached the target: check experiment logs/events for the injection step, confirm the target pod/container was matched, and look for the expected symptom in at least one signal. If no symptom appears, treat it as an experiment-validity question before concluding resilience.