Canary Release With a Broken Metrics Gate
A team uses a progressive delivery controller to canary a new version of payments-api. The rollout is defined with steps: 10% traffic for 5 minutes, then 50%, then 100%. The analysis template queries a Prometheus metric http_request_success_rate and requires it to stay above 0.99. During the 10% step, the success rate reads 0.995 and the rollout proceeds to 50%. At 50%, the success rate drops to 0.97, but the controller does not abort because the metric query returns 'no data' for one of the two canary pods (its sidecar hasn't registered yet), and the controller treats missing data as a pass. Meanwhile, the stable version's pods are still serving 50% of traffic and their success rate is 0.999. The team only notices because a customer complains.
1. Why did the success-rate gate fail to catch the regression at 50%, and what is the flaw in treating 'no data' as a pass?
Read the worked answer
Two compounding issues:
Aggregation masks the canary. If the metric is summed/averaged across all pods (canary + stable), the 0.97 canary is diluted by the 0.999 stable pods. At 50/50, the blended rate is ~0.9845 - still below 0.99, so a correct aggregate would have caught it. But if the query filters by
version=canaryand one canary pod has no data, the remaining single pod's rate may be 0.995, passing the gate.Missing data treated as pass. The controller's
failureLimit/inconclusiveLimithandling matters. Ifno datais counted as success, a pod that never reports (sidecar not registered, scrape gap, or a crashed exporter) is silently excused. That's dangerous: the absence of telemetry is exactly when you should stop, not proceed. The canary pod with no data may be the one that's failing.
The fix: treat missing data as inconclusive (which should pause or abort after N consecutive inconclusive results), and scope the query strictly to canary pods with an explicit increase() over a window so a single scrape gap doesn't zero the result.
2. Describe the corrected analysis configuration and a rollout-safety change.
Read the worked answer
Query: Scope to the canary ReplicaSet label and use a rate over a window longer than the scrape interval, e.g. sum(rate(http_requests_total{job="payments-api",version="canary",code!~"5.."}[2m])) / sum(rate(http_requests_total{job="payments-api",version="canary"}[2m])). Requiring at least 2 successful scrapes before evaluating avoids single-point noise.
Gate semantics: Set successCondition: result[0] >= 0.99, failureLimit: 1, and inconclusiveLimit: 2. Missing data → inconclusive; two consecutive inconclusive results → abort. This ensures a silent canary pod halts the rollout rather than sailing through.
Rollout-safety change: Add a prePromotionAnalysis that runs the gate against the canary before shifting traffic to 50%, and add a manual approval gate between 10% and 50% for a payments service. Also add a second metric (error rate / latency p99) so a single metric can't be the only signal. Finally, verify the sidecar's readiness gates the canary pod's inclusion in the Service so it can't receive traffic before it can report metrics.