Autoscaling Instances Revert to the Previous Release
Corvus Health runs a patient portal on 8 Compute instances behind an OCI load balancer, spread 3/3/2 across three ADs in the Ashburn region. An instance pool with autoscaling (min 6, max 14, CPU target 60%) adds capacity during clinic hours. Releases go through an OCI DevOps deployment pipeline that targets an instance group with a rolling strategy, batch size 2, and a 5-minute wait between batches. After each of the last three releases, support found 1-3 instances running the previous release, and the load balancer reported those instances as healthy. During one release a scale-out event added 4 instances mid-deployment, the stage ran past its 30-minute timeout, and the release was marked failed while 6 of 8 instances had already been updated. Instances have no public IPs; the DevOps agent is installed on each one.
1. Why do instances with the previous release keep appearing after a successful deployment, and how do you fix it?
Read the worked answer
Autoscaling launches instances from the instance configuration, and the deployment never updates it. The configuration captures a custom image and/or user data that pins the application version at the moment the configuration was created. The rolling deployment updates the eight running instances; it does not rewrite the template. Every scale-out afterwards therefore plants a fresh instance running the old release. Those instances pass the load balancer health check because the health endpoint reports liveness, not version.
Fixes:
- Make the instance configuration part of the release. Produce a new custom image as a build artifact, and have the deployment (or an immediately preceding pipeline stage) update the instance configuration to reference the new image or new artifact version before the rolling stage begins. Instances launched mid-rollout then match the target version.
- Alternatively have instances pull a version at boot from Object Storage - but if the pointer is mutable this recreates the
latest-tag problem, so pin the version in the instance configuration. - Verify version, not just liveness. Have the application's health response include the running build SHA, and make the post-deploy stage query all instances through the load balancer (or the instance inventory) and fail unless every instance reports the target SHA.
- Bake the DevOps agent into the image so a newly scaled instance is immediately deployable and does not sit outside the instance group.
2. Why did the stage hit its 30-minute timeout during the scale-out, and how do you make rolling releases predictable?
Read the worked answer
The batch math was already tight and the scale-out made it unbounded. Eight instances at batch size 2 and a 5-minute wait is four batches, roughly 20 minutes before per-instance agent work and load-balancer drain are counted - a 30-minute ceiling with almost no headroom. Adding four instances mid-deployment extends the work and changes the target set while the stage is running, so the run crosses the timeout with only part of the fleet updated.
What to change:
- Right-size the batch against real capacity. Batch 4 with a 5-minute wait is two batches, but only if the remaining instances can carry peak load without breaching CPU headroom. Verify with the last month's peak CPU per instance rather than guessing.
- Bound the deployment window. If you keep batch 2, raise the stage timeout to around 45 minutes and add a per-instance time limit so one unresponsive agent cannot consume the whole window.
- Freeze scale-out for the deployment window (pin desired capacity or suspend the autoscaling policy) so the target set is stable; let the rolling logic handle capacity during that period, and schedule releases outside clinic hours.
- Make the wait a verification step, not a sleep - query the version endpoint on updated instances before the next batch starts.
- Set the load balancer drain/connection timeout longer than the wait interval. If the 5-minute wait is shorter than the drain timeout, two batches can overlap in the backend set and users hit instances whose application has just been stopped, which shows up as intermittent 502s during releases rather than as a deployment failure.