The 09:00 Spike That Autoscaling Never Catches

The spike is a queueing problem, and the autoscaler is reacting too late to a leading indicator it does not measure.

Chain of events:

1. Lag exceeds the event. The burst lasts 10 minutes. Each new instance needs 3 minutes 40 seconds of image pull plus boot before it takes traffic, and the scaling policy needs minutes of sustained high CPU before it even fires. By the time capacity arrives, the burst is largely over - the group scales into the tail of the event, then scales back in and is cold for tomorrow.
2. CPU is the wrong signal. Image processing is queue-based. The thing that predicts user pain is queue depth and per-request wait time, not average CPU. Average CPU is also smoothed across 12 instances, so a saturated subset looks mild.
3. No admission control. A 40,000-image batch treated identically to interactive traffic means batch work can consume all capacity and push interactive requests into the queue.

Changes, in order of effect:

• Scheduled scaling ahead of the predictable 09:00 burst - the workload is not random; treat it as a timetable. Add minimum capacity at 08:45 and remove it at 09:30.
• Scale on a leading indicator: queue depth or requests-per-target, or target p95 latency, rather than average CPU. Keep a CPU policy as a secondary safety net.
• Cut launch time: pre-baked AMI or image cache on nodes, smaller base image, and a warm pool so instances are already booted. A warm pool converts 3m40s into seconds and is usually the cheapest of the three fixes.
• Separate the batch lane with its own queue and concurrency limit, so batch throughput degrades before interactive latency does.

Verification: replay the 40,000-image burst in a load test; measure time to first scale-out action and time to serve under the latency SLO. Target: p95 stays under 500 ms.

The hazard is correlated reclamation. A spot group spread across one instance type and one availability zone can lose a large fraction of capacity at once, and the remaining on-demand 60 percent must absorb both the missing capacity and the traffic it was serving.

What makes it safe:

• Diversify across several instance families, sizes, and availability zones, and use the allocation strategy that picks from the deepest pools. Diversity is the actual defence; a single instance type is a single point of failure regardless of the discount.
• Handle the two-minute notice. On interruption, deregister from the load balancer, drain in-flight requests, checkpoint or requeue unfinished work, then terminate cleanly. Without the drain, every reclamation produces 502s for requests mid-flight.
• Keep the on-demand baseline at the sustained peak, not at the average. Spot should cover the top slice, not the floor.
• Replenish automatically: the autoscaling group's rebalancing and instance-refresh behaviour should replace reclaimed capacity, and the warm pool should make replacement fast.

Failure mode if you skip this: a reclamation event removes 40 percent of capacity in under two minutes. The survivors are already at the autoscaling target from the 09:00 pattern, so they saturate, queues grow, p95 latency breaches SLO, and the autoscaler tries to launch replacements that also need 3m40s to boot. The two failures compound - capacity loss plus slow replacement - which is exactly the shape of a long, noisy outage rather than a blip.

Cheaper alternative to consider: if the batch workload tolerates delay, run it on spot and interactive traffic on on-demand, rather than mixing both on every instance. That makes reclamation a batch-latency problem rather than a customer-facing one.

The current alert fires on the outcome (p95 breach) after customers are already affected. Add leading indicators:

• Queue depth / age of oldest item with an alert on sustained growth, not on an absolute spike - a short spike is normal, a rising trend is not.
• Scaling activity: alert if the group is at maximum desired capacity, or if a scale-out takes longer than the warm-pool target.
• Time-to-serve p95 for the batch lane, tracked separately from interactive latency.
• Spot interruption rate and post-drain error counts, so reclamation shows up as a metric rather than a support ticket.

Verification plan - three parts, because each tests something different:

1. Replay the real burst in a load test at 09:00 volume and 1.5x, and measure p95 latency, scale-out timing, and whether scheduled capacity was already in place before the burst began. A fix that only works because the test was gentle is not a fix.
2. Chaos the spot tier: deliberately terminate 40 percent of spot instances during peak and confirm draining produces no 5xx and replacement completes inside the warm-pool target.
3. Run three consecutive live weekdays and compare p95 and page volume against the previous baseline. Three days covers the day-of-week variance in the marketing batch.

Also check the cost delta. Scheduled capacity and warm pools both cost money at idle. Confirm the increase is bounded and that the nightly 02:00–05:00 job still completes inside its window - moving instances around can silently shift contention onto the thumbnail rebuild, which competes for the same database.