The Blue/Green That Wasn't
A team deploys a containerized API to ECS Fargate behind an ALB using a blue/green strategy with CodeDeploy. The service handles 800 requests/second. A release introduces a memory leak that only manifests after ~10 minutes under load. CodeDeploy is configured to shift 10% of traffic, wait 5 minutes, then shift the rest. Health checks pass because the leak takes longer than 5 minutes to cause failures. The team discovers the issue only when p99 latency degrades 40 minutes after full cutover. Rollback requires redeploying the previous task definition, which takes 15 minutes.
1. Explain why the current deployment configuration failed to catch the defect, and what deployment controls would have.
Read the worked answer
The deployment's bake time (5 minutes) is shorter than the defect's latency (10 minutes). Traffic shifting is a mechanism for limiting blast radius; it only catches defects whose symptoms appear within the observation window. Because health checks only look at process liveness and the leak manifests as gradual memory growth, the green environment looked healthy at the moment of full cutover.
Controls that would have caught it:
- Extend the bake time to exceed the defect's time-to-symptom - e.g., 20-30 minutes at 10% before full shift.
- Add a CloudWatch alarm-based rollback on memory utilization (ECS
MemoryUtilization) and p99 target response time, wired into the CodeDeploy deployment group so a breach triggers automatic rollback. - Canary at a smaller fraction (e.g., 1-5%) for longer, so fewer users are affected while the signal accumulates.
- Pre-deployment load test in a staging environment at production scale for at least the leak's time constant.
The general principle: the observation window must be longer than the slowest failure mode you care about, and rollback must be automatic and fast.
2. The team wants to eliminate the 15-minute rollback time. What architectural changes make rollback near-instant?
Read the worked answer
The 15-minute rollback is because rollback means redeploying rather than repointing. To make it near-instant:
- Keep the blue environment warm during the bake period instead of terminating it. Rollback then means shifting the ALB listener back to blue - seconds, not minutes. CodeDeploy blue/green already does this if you don't tear down blue prematurely.
- Pin immutable image tags (digest, not
latest) so the previous task definition is guaranteed to pull the exact prior image. - Decouple schema changes from code - use expand/contract migrations so the old and new versions can run concurrently and rollback never requires a database revert.
- Pre-warm capacity in the green target group so cutover doesn't wait on task startup.
- Automate the rollback trigger via alarms so the decision-to-action latency is bounded by the alarm evaluation period, not human response.
With warm blue, rollback is bounded by ALB target group weight change propagation (seconds), meeting a much tighter RTO.