Works in Dev, Blocked in Prod
A Harness Chaos Engineering experiment runs successfully in dev but fails in prod before any fault is injected. The experiment uses a ChaosHub pod-delete template, a Kubernetes connector, and a service account. In dev, the service account has a RoleBinding to a ClusterRole allowing pods, pods/log, and events. In prod, the same service account name exists, but its RoleBinding is scoped to the application namespace, not the chaos runner namespace, and it omits pods/log. The org also has a Chaos Guard policy that blocks experiments where the environment tag is 'prod' unless an approval is present. The run log shows 'pre-flight check failed: cannot create chaos runner' and no fault pod appears. The experiment is scheduled during a change freeze; the on-call engineer has 30 minutes. The team wants to disable Chaos Guard for the prod project to get the experiment running, then fix RBAC later.
1. Diagnose the failure in the likely order of checks, and explain what evidence would distinguish policy denial from RBAC.
Read the worked answer
Harness evaluates policy and pre-flight conditions before fault injection. First check the Chaos Guard audit trail for a denial tied to the prod environment tag or a missing approval. If no policy denial exists, the error points to runner creation in the target namespace. Distinguish the two with evidence: a guard denial appears as a policy evaluation/denial entry, while an RBAC problem appears as a Kubernetes API 403 from the service account. Verify effective permissions with kubectl auth can-i create pods --as=system:serviceaccount:<ns>:<sa> in the chaos runner namespace and can-i get pods/log. The prod RoleBinding is scoped to the application namespace and lacks pods/log, so it does not authorize runner creation where the runner is created. Also confirm the connector points at the correct cluster and namespace.
2. Propose a fix and a validation plan that does not disable Chaos Guard.
Read the worked answer
Do not disable the guard. Use an approval or a time-bound exception scoped to the experiment, environment, user, and duration. Fix RBAC with a dedicated prod chaos service account and a RoleBinding in the chaos runner namespace granting only the verbs the runner needs, including pods/log for log collection. Validate with a no-op or single-pod canary experiment in prod: 1 pod, 1-minute duration, explicit abort conditions, and a change record. Confirm the runner is created, logs are collected, and the guard audit still shows the approval path. If the change freeze blocks RBAC changes, rehearse in a prod-like staging namespace first; do not weaken the guard globally to work around an authorization gap.