Empty Vector, Green Report
A steady-state probe for a Harness chaos experiment runs a Prometheus query every 30 s: sum(rate(http_requests_total{service='payments',code=~'5..'}[1m])). The success criterion is 'value < 0.01'. A platform upgrade renames the label from service to service_name and changes the metric export path. The query now returns an empty result. The probe evaluator treats an empty vector as zero and marks the probe passed for the entire 15-minute experiment. The experiment deletes 3 of 8 payment pods. After the run, the team sees a 2.5% 5xx rate in the service mesh dashboard, but the Harness experiment report is green. The team has a 99.9% availability SLO and a 5% error budget remaining. They want to add a second probe for latency and rerun.
1. Why is the green report invalid? Name the failure mode and the consequence for the experiment's conclusion.
Read the worked answer
The query returns an empty vector because the label changed from service to service_name. The evaluator maps that empty result to zero, and 0 is less than 0.01, so the probe passes. This is a false pass: missing data was treated as success. The experiment's conclusion is invalid because the steady-state check did not observe the system at all. The service mesh shows 2.5% 5xx, so the fault very likely violated the hypothesis. The consequence is worse than a failed experiment: the team may close a risk or approve a change based on a green report that never measured the relevant signal.
2. Design a corrected probe set that fails closed on missing data.
Read the worked answer
Update the query to use service_name='payments' and keep the 5xx rate criterion. Add an absent(http_requests_total{service_name='payments'}) companion probe whose success value is 0; if the series disappears, it returns 1 and the probe fails. Configure the evaluator so probe timeout or no data is failure, not success. Add a latency probe using histogram_quantile for p99 and an availability/error-budget probe. Establish a 30-minute baseline before the experiment. Success criteria: 5xx <0.1%, p99 <300 ms, and no missing series. Test the probes against a known bad canary in staging before trusting them in a chaos run.
3. Given 5% error budget remaining, should they rerun immediately? What blast-radius and sequencing controls apply?
Read the worked answer
Not immediately. First fix and validate the probes in staging, and calculate whether the observed 2.5% 5xx for 15 minutes already consumed a material share of the remaining budget. Then run a limited production experiment: 1 of 8 pods, 5 minutes, abort if 5xx >0.5% for 1 minute or if error-budget burn exceeds 1% of the remaining budget. If the budget is a hard constraint, run in a canary namespace instead. The low budget changes the decision more than the technical fix does: a smaller blast radius and a fail-closed probe set are prerequisites, not optional improvements.