Instance Pool Flapping Caused by Health Check Warm-Up Mismatch
An instance pool web-pool runs 12 VM.Standard.E4.Flex instances (4 OCPUs, 32 GB) behind a flexible load balancer in a regional VCN. Autoscaling is CPU-based: scale out at 60% for 5 minutes, minimum 6, maximum 24. The load balancer health check is HTTP GET /health on port 8080, expecting HTTP 200, with a 10-second interval and 3 failures. The application listens on port 80; its /health endpoint on 8080 returns 503 until a 90-second JVM warm-up completes. The instance pool's health check is configured to use the load balancer health status, with a 60-second grace period. During a spike, the pool adds 6 instances. Each new instance is marked unhealthy by the load balancer at about 60 seconds, and the pool terminates and replaces it. The cycle repeats; existing instances serve traffic but latency rises, and the pool never reaches a stable size.
1. Explain why the pool terminates new instances even though the application is starting correctly. Why would the load balancer alone not cause termination?
Read the worked answer
The instance pool is using the load balancer health status as its own health check. The load balancer marks a backend unhealthy after 3 failures at 10-second intervals, so a /health endpoint returning 503 for roughly 30 seconds is enough to be considered unhealthy. The pool grace period is only 60 seconds, which is shorter than the 90-second JVM warm-up. At 60 seconds, the pool sees the load balancer health status as unhealthy and replaces the instance before it can become healthy. The load balancer alone would only stop sending traffic to that backend; it would not terminate the instance. The termination is the pool's replacement action, and it creates the loop.
2. What changes would you make to stabilize the pool? Give at least three options and their trade-offs.
Read the worked answer
Option 1: increase the instance pool grace period to 180 seconds, comfortably above the 90-second warm-up plus margin. Trade-off: genuinely failed instances stay in the pool longer before replacement. Option 2: change the health endpoint semantics so /health returns 200 as soon as the process can accept connections, and use a separate /ready endpoint for load balancer traffic after warm-up. This separates liveness from readiness. Trade-off: requires application changes and a second check. Option 3: use the compute health check for pool replacement and keep the load balancer health check only for traffic steering. Trade-off: the pool will not replace an instance whose application is hung as quickly, so you need separate application monitoring. A fourth option is to pre-warm instances with a custom image or startup script, but that adds image maintenance. The best combination is usually a readiness endpoint plus a grace period longer than warm-up.
3. How would you validate the fix without waiting for a production traffic spike?
Read the worked answer
Use a canary scale-out: manually increase the pool by one instance during a low-traffic window and watch the instance lifecycle, load balancer backend health, and application logs. Confirm that the instance stays in the pool past the grace period and that the load balancer only sends traffic after readiness. Repeat with a deliberate application failure to confirm the pool still replaces a genuinely bad instance. Check autoscaling history and pool events to verify no replacement loop. If the pool supports rolling updates, test the new health check settings in a staging compartment with the same image and startup time before changing production.