A Scale Set That Scaled Out but Did Not Get Faster

Cause 1: CPU was the wrong signal, or the wrong instance metric. Scaling on average CPU across the set hides the tail. If the latency is driven by a downstream dependency (database, cache, thread pool exhaustion, GC), adding CPU headroom does nothing. Start with Application Insights dependency durations and VM insights per-instance CPU, memory, and network to find the actual saturation point before touching the rules. Autoscaling on a custom metric or a queue length is often the correction.

Cause 2: the rules reacted too slowly. Six instances in 25 minutes implies roughly four scaling steps, each waiting a 10-minute evaluation window, a 5-minute cool-down, plus provisioning and boot. That is a 15-20 minute lag behind demand. For scale-out, shorten the window to 5 minutes, drop the cool-down to 2-3 minutes, and scale by 2-3 instances per step. Pair it with a scheduled profile that pre-warms capacity before the known Monday peak, which removes the reaction delay entirely.

Cause 3: instances were not actually serving. 40% of traffic to two instances is the signature of a small healthy backend set, not of load balancing being unlucky. The custom script extension now failing or racing on most instances is the likely link: instances that never became ready stayed out of rotation or failed health probes, leaving few endpoints. Check backend pool health, probe status, and the extension's status message and exit code per instance. Fix the agent installation, add a readiness probe that only passes when the app is warm, and confirm healthy-instance count equals instance count after every scale-out.

Also check load balancer distribution. The default is a 5-tuple hash with no stickiness; traffic from a small number of source IPs behind a corporate proxy can concentrate on few backends. If sessions are involved, affinity changes the diagnosis entirely.

Root cause. Sessions are held in the process or in instance-local storage, so terminating an instance destroys them. Autoscale scale-in is an abrupt operation: the platform picks instances according to the scale-in policy and deletes them, and the load balancer simply stops sending new connections. Nothing in the current configuration drains or warns the application.

Fix 1: remove the instance affinity from the design. Move session state to a shared store such as Azure Cache for Redis (or make the API stateless with tokens). Once a request can be served by any instance, terminating one is invisible to users. This is the only fix that also survives host failures, patching, and zone outages, and it should be the primary answer.

Fix 2: drain before termination. Enable termination notifications on the scale set so the platform delivers a Scheduled Event to the instance before deletion, with a configurable delay (up to 15 minutes) set on the scale set's scheduled-event profile. The application subscribes to the event, stops accepting new work, finishes in-flight requests, and exits. This requires application support - the event is cooperative, not enforced - so it is a complement to Fix 1, not a replacement.

Fix 3: slow the scale-in path down. Keep aggressive scale-out and conservative scale-in: longer cool-down, decrease by one instance per step, higher scale-in threshold. Set the scale-in policy deliberately (for example OldestVM or NewestVM) rather than leaving the default, and test it in a non-production set.

Fix 4: raise the floor. A minimum of 2 in a three-zone deployment gives no zone redundancy and no headroom to drain. A minimum of 3, one per zone, is a more defensible baseline for a customer-facing API.

What Spot does and does not give you. Spot VMs are capacity from surplus Azure inventory at a large discount, and they can be evicted with 30 seconds' notice when Azure needs the capacity back or when the price exceeds the maximum you set. That is the opposite of what a customer-facing API on a latency-sensitive workload needs on its primary serving tier, especially one that already lost user sessions during scale-in.

The specific risks in this design. Evictions are not correlated with traffic; a single eviction wave can remove 60% of capacity while a scale-out is still 15 minutes behind, which is precisely the failure mode already observed. The 30-second notice is shorter than the application's ability to drain, so evictions become session losses. Spot also constrains the SKU and zone combination available, so instances may not land evenly across three zones, undermining the zone-redundancy design.

What I would recommend instead. Keep the customer-facing scale set on regular VMs with a floor of 3, one per zone. Put genuinely interruptible work - batch jobs, report generation, image transcoding, CI runners - on a separate Spot scale set or separate Spot VMs behind its own capacity logic, with eviction policy Delete so the disk goes away with the VM and a maximum price of -1 where the workload tolerates market rates. If cost reduction is the actual driver, attack it with a savings plan or reserved instances on the baseline capacity (the floor of 3 runs 24/7 and is a good reservation candidate), right-size the SKU from VM insights, and add scheduled scale-in outside business hours.

If Spot must be used in the serving tier, cap it well below 60%, isolate it in its own scale set so the regular set guarantees the floor, set a larger minimum on the regular set than the expected peak, and make eviction handling explicit with termination notifications before go-live.