Blue/Green Cutover With a Stateful Backend
A team is moving a monolith from EC2 into a new blue/green deployment on ECS Fargate behind an ALB. The app keeps user sessions in memory and writes uploaded files to local disk (/var/uploads). Traffic is ~2,000 requests/minute at peak, and the business cannot tolerate more than ~30 seconds of errors during cutover. The current EC2 fleet will remain as the "blue" environment for two weeks as a rollback path. The team's first cutover plan is: deploy green, shift 100% of ALB traffic to green, terminate blue. During a staging rehearsal, users were logged out on cutover and files uploaded to blue were missing on green.
1. Diagnose why the rehearsal failed and redesign the cutover to meet the 30-second error budget.
Read the worked answer
Two design flaws, both about state living on the instance:
- In-memory sessions. Shifting traffic to green means green has no session state, so every user is logged out. Fix by externalizing sessions (ElastiCache/Redis or a signed cookie) before the cutover, or by using sticky sessions during a transition - but stickiness alone doesn't survive the blue termination, so externalizing is the durable fix.
- Local disk uploads. Files written to blue's
/var/uploadsdon't exist on green. Fix by moving uploads to S3 (or EFS if POSIX semantics are required) and having both environments read/write the shared store. This must be done while blue is still serving so no data is stranded.
Cutover redesign to hit 30 seconds:
- Pre-warm green and verify health checks pass before any traffic moves.
- Use weighted ALB target groups to shift traffic gradually (e.g., 10% → 50% → 100%) rather than an instant 100% flip. This bounds the blast radius of any residual bug and keeps errors well under 30 seconds of exposure.
- Keep blue registered and healthy (scaled down but serving) for the two-week rollback window; rollback is a weight shift back, not a redeploy.
- Only terminate blue after the rollback window closes and you've confirmed no writes are still landing on local disk.
The rehearsal failure was a state problem, not a deployment problem - no amount of blue/green tooling fixes state that lives on the instance.
2. The team asks whether they can skip externalizing sessions and just use ALB sticky sessions for the two-week window. Advise them.
Read the worked answer
Sticky sessions are a bridge, not a solution, and only safe with conditions.
- Stickiness binds a user to a target for the cookie's lifetime. During a gradual weight shift, a user stuck to blue stays on blue until their cookie expires or blue is removed - so "100% weight to green" doesn't actually mean 100% of users are on green. When you finally drain blue, every still-stuck user is logged out at once. That converts a controlled cutover into a delayed, synchronized failure - the opposite of the 30-second goal.
- Stickiness also concentrates load: if a few heavy users land on one target, that target bears disproportionate traffic, which matters at 2,000 rpm.
If they insist for the two-week window, require: (1) short cookie TTLs so users migrate naturally, (2) monitoring of how many sessions remain pinned to blue before termination, and (3) a hard rule that blue is not terminated until pinned sessions approach zero. But the honest recommendation is to externalize sessions now - it's the only approach where "traffic is on green" and "users are on green" are the same statement.