Oracle Cloud Infrastructure Architect Associate: The autoscaling policy that kept its promise and still failed

Do the arithmetic first. Scaling from 4 to 12 in steps of 2 needs four scale events, each followed by a 300 s cool-down, so the last instance is not even requested before roughly 20 minutes - then it needs minutes to boot. Reaching 8 within 15 minutes is the policy working exactly as written. It is too slow, not broken, and "increase max" would not have changed anything.

Then check whether it was also capacity. A single availability domain plus a 4-OCPU Flex shape is where "out of capacity" shows up silently: instances sit in provisioning or fail, and the pool never reaches max. Check the pool's work requests, and check the compute core service limit for that shape family in that AD. Spreading the pool over three ADs with a regional subnet removes a single point of capacity failure as well as a single point of infrastructure failure.

Ranked fixes:

1. Pre-scale on a schedule before known events (a schedule-based policy that sets a higher minimum), because no reactive policy beats a known spike.
2. Increase the step size (4 instead of 2) and cut the cool-down to 120 s - the point of the cool-down is to avoid thrash, and 300 s is generous when boot takes minutes anyway.
3. Remove boot time from the critical path: bake the application into a custom image so the instance is serving within a minute instead of installing packages via cloud-init.
4. Raise the compute limit and spread across ADs.

Do not lower the CPU threshold to 60% and call it fixed; that just makes the pool noisy and expensive at normal load.

What happens on removal. When the pool scales in, the chosen instances are terminated. There is no drain step. The load balancer only stops using a backend after its health checks fail - roughly 3 × 10 s, about 30 seconds - and connections accepted in that window land on a socket that is going away. Two instances terminated together, plus client retries, lines up with the observed 90 seconds.

Why it is worse than it looks. If HTTP sessions live in the JVM, every user pinned to those instances is logged out or loses a cart, and retries hit the remaining instances with a cold cache. The 502 is the visible symptom; lost session state is the real damage.

Changes, in order of value:

1. Make the tier stateless. Move sessions to a shared store or signed tokens so any instance can serve any request. This is the actual fix; everything below only shortens the window.
2. Raise the minimum from 3 to a floor that covers overnight traffic so scale-in rarely fires at all. Autoscaling that fights a predictable daily curve is costing you money in incidents, not saving it.
3. Use a schedule-based scale-in that removes one instance at a time during the quietest hour, instead of a CPU policy that removes two the moment the average dips.
4. Handle SIGTERM in the application: stop accepting new work, finish in-flight requests, exit. This buys graceful termination even without a platform drain.
5. Tune the health check to fail faster (5 s interval, threshold 2) so the backend is withdrawn sooner.

Also check whether anything stateful hides on those instances - cron jobs, local caches, log files that are not shipped - because those disappear with the instance too.

Where the time goes. The load balancer needs about 30 seconds (3 × 10 s) to mark a bad backend, so four to five minutes is boot and application start, not health-check tuning. With a stock image the first boot runs cloud-init: package installs, configuration management, JVM warm-up. During that time /health either refuses connections or returns a non-200, correctly.

Diagnose before fixing. Pull the instance console history and the cloud-init output; time each phase. If package installation dominates, the answer is a custom image. If the application itself takes three minutes to warm a cache, the answer is a health endpoint that reports readiness (dependencies reachable, cache warm) separately from liveness, plus a threshold that tolerates warm-up.

Fix:

1. Build a custom image with the application pre-installed and only environment-specific configuration left to cloud-init. This typically cuts start-up to under a minute and also makes scaling faster during incidents.
2. Keep /health cheap and deterministic; do not let it query the database on every probe.
3. Reduce the unhealthy threshold only after boot is genuinely short, so you do not mask real failures.

Rolling it out. Create a new instance configuration with the new image and update the pool. Instances already running keep the configuration they were launched with, so nothing changes until they are replaced. Replace in batches - detach or terminate a couple at a time, wait for them to pass health checks before continuing - and never combine the rollout with a scaling event or a deployment.