Canary Deployment of a Changed Orchestrator
A multi-agent research assistant has been stable for months. The team is deploying a new orchestrator prompt that improves synthesis quality but changes how sub-tasks are delegated, plus an upgraded model for one specialist. Deployment is a single production slot; there is no staging environment that mirrors production traffic. Rollback currently means redeploying the previous container image, which takes about 20 minutes and also rolls back the specialist model. The team proposes a full cutover at 2 a.m. and monitoring error rates and CSAT for 24 hours. Last quarter a similar cutover caused a 6-hour degradation that was only caught when a customer complained, because error rates stayed flat while answer quality dropped.
1. Critique the deployment plan and propose a safer rollout that fits the single-slot constraint.
Read the worked answer
The plan fails on three counts:
- Full cutover = maximum blast radius. Every user gets the new orchestrator and the new model simultaneously; if quality drops, 100% of traffic is affected.
- The monitoring is blind to the actual risk. Last quarter's incident showed error rates stay flat while quality degrades. Error rate and CSAT (a lagging signal) will not catch a synthesis-quality regression within 24 hours.
- Rollback is slow and coarse. A 20-minute image rollback that also reverts the specialist model couples two independent changes and lengthens time-to-recovery.
Safer rollout within a single slot:
- Decouple the two changes. Ship the orchestrator prompt change and the specialist model upgrade separately, so a regression can be attributed and reverted independently.
- Traffic-split canary via routing, not slots. Use an orchestrator-level feature flag / routing layer to send a small slice (e.g., 5–10%) to the new version while the old version serves the rest. This does not require a second slot.
- Quality-based gates, not just error rates. Define online metrics that detect quality drift (groundedness, task success from sampled LLM-judge or human review) and set thresholds that automatically halt the ramp.
- Ramp on evidence: 5% → 25% → 100% only as quality metrics hold, with the ability to flip the flag back instantly (seconds, not 20 minutes).
Keep the old image warm so rollback is a flag flip plus, if needed, a fast revert.
2. What should the canary's success criteria and automatic rollback triggers be, given that error rates are known to be uninformative?
Read the worked answer
Define criteria before the rollout, on metrics that actually move when quality moves.
Primary quality gates (sampled live traffic, compared canary vs. control):
- Task success / groundedness from an LLM judge that has been validated against human labels, with a confidence interval. Halt if the canary's lower bound falls below the control's.
- Delegation correctness: did the new orchestrator produce the intended sub-task decomposition? Compare against a labeled sample.
- Human escalation / correction rate as a secondary, slower signal.
Guardrail metrics:
- Latency (p95) and cost per request - a new orchestrator that fans out more will show up here first.
- Hard error rate - keep it as a guardrail even though it is known to be insensitive; it catches crashes, not quality.
Automatic rollback triggers:
- Any guardrail breach beyond threshold (e.g., p95 latency +30%, error rate above baseline).
- Quality metric statistically below control on the sampled set.
- A single high-severity safety/groundedness failure (e.g., fabricated citation) in the canary sample - fail closed on severe individual failures even if aggregate looks fine.
Because error rates are uninformative, the rollout must be gated on sampled quality evaluation, with a pre-agreed sample size large enough to detect a meaningful drop, and rollback must be a fast flag flip rather than a 20-minute redeploy.