Autoscaling Loop Behind an ALB During a Flash Sale
An e-commerce site runs an Auto Scaling group behind an Application Load Balancer in two availability zones. During a flash sale, the group scales from 8 to 40 instances in 10 minutes. Each instance needs 90 seconds to start and warm its JVM cache. The ALB health check is HTTP GET /health every 10 seconds, timeout 2 seconds, unhealthy threshold 2; /health checks a database connection and returns 500 if the pool is exhausted. The RDS database allows 100 connections, and each instance opens 10. Within 15 minutes, 28 instances are marked unhealthy and replaced; CPU on the remaining instances rises, and the cycle repeats. The team proposes increasing the maximum group size from 40 to 80 and lowering the health check interval to 5 seconds.
1. What is the root cause of the replacement loop, and why will increasing maximum size and shortening the health check interval make it worse?
Read the worked answer
The loop is caused by a deep health check combined with no startup grace. /health depends on the database, and each new instance opens 10 connections. At 40 instances that is 400 connections against a 100-connection limit, so the database refuses connections, /health returns 500, the ALB marks instances unhealthy after two failed checks, and the ASG replaces them. Replacements repeat the same connection storm. Raising the maximum to 80 increases the storm; lowering the interval to 5 seconds makes false unhealthy decisions faster. This is negative scaling: adding capacity consumes a shared resource and removes healthy capacity. The fix is not a larger ASG or a more aggressive health check; it is to remove the database dependency from the liveness check and control connection concurrency.
2. Recommend specific changes to the health check, scaling policy, and database connection handling.
Read the worked answer
Change the ALB health check to a shallow /healthz that returns 200 after JVM initialization and does not query the database. Use interval 30 seconds, timeout 5 seconds, healthy threshold 3, unhealthy threshold 3, and a deregistration delay that drains connections. Add an ASG lifecycle hook or instance warmup of at least 180 seconds so new instances are not judged during JVM warmup. Use a warm pool if startup is consistently 90 seconds. For the database, put RDS Proxy or PgBouncer in front of RDS, cap each instance pool below 10, add retry with backoff, and raise the database connection limit only after measuring. Use step scaling with a cooldown and pre-scale before known sales rather than relying only on reactive scaling.
3. Under what conditions should the team not rely on horizontal autoscaling for this workload?
Read the worked answer
Do not rely on horizontal autoscaling when the bottleneck is a shared stateful dependency such as the database, when startup takes longer than the traffic spike, or when each instance adds a fixed connection cost. If the database is the constraint, adding application instances makes the failure worse. In those cases use pre-scaling for known events, caching to reduce database reads, read replicas for read-heavy paths, queue-based load leveling, and vertical scaling or a connection proxy. Horizontal autoscaling is appropriate for stateless web tiers with fast startup and a backend that can absorb the added connections. The team should test the full scale-out path, including database connection limits, before the next sale.