Autoscaling an Inference Endpoint Under Bursty Traffic

Three independent mismatches compound:

1. Wrong signal. CPU utilization on GPU nodes is a poor proxy for GPU saturation. GPU inference is often memory-bandwidth or SM-bound; CPU can sit at 30% while the GPU queue backs up. Scaling on CPU means the trigger fires late or never, even as latency climbs. The correct signal is queue depth, GPU utilization, or p99 latency / requests-in-flight per replica.
2. Cooldown longer than the event. Scale-up cooldown is 5 minutes; spikes last 2-4 minutes. By the time the first new node is permitted, the spike is over. The system can never react within the event window.
3. Cold start exceeds the spike. Node provisioning takes 90 seconds - most of a 2-4 minute spike - and the 1.8 GB model load is the dominant part. Even perfect triggering arrives too late for short bursts.

Compounding: baseline headroom is 6 × 500 = 3,000 req/s, exactly the peak. So the system is sized for the peak but with zero margin for the ramp, and any single-node degradation pushes it over. The gateway's 2-second timeout then converts latency into dropped revenue.

Diagnosis summary: signal blind to the real bottleneck, control loop slower than the disturbance, and capacity that cannot be added faster than demand arrives. Each needs a different fix; tuning one alone will not help.

Separate steady-state capacity from burst absorption, and make degradation explicit.

Burst absorption (buy time, don't scale):

• Keep a warm pool of pre-loaded replicas (model already in GPU memory) sized for the observed flash-sale peak. Pre-loading removes the 90-second cold start from the critical path. Cost is idle GPU time; price it against dropped transactions.
• Queue with bounded wait: admit requests into a queue with a deadline of ~1.5 s so responses return before the 2 s gateway timeout. Reject-at-deadline rather than let requests time out invisibly.

Faster scaling for sustained load:

• Scale on requests-in-flight per replica and GPU queue depth, not CPU. Target a utilization that leaves headroom for the ramp.
• Shorten cooldown to 30-60 s for scale-up (keep a longer, damped cooldown for scale-down to avoid flapping).
• Shrink cold start: bake the model into the image or mount it from a fast local cache, use smaller/quantized model variants for burst replicas, and pre-pull images on standby nodes.

Deterministic fallback (compliance):

• When queue deadline or capacity is exceeded, return a rule-based fallback decision (e.g., static risk tier by amount and velocity) with an explicit degraded=true flag and reason code. This is deterministic, auditable, and never a silent drop.
• Log every fallback with the request ID so compliance can reconcile.

Load shedding order: shed lowest-value or lowest-risk traffic first; never shed uniformly, since that randomizes which transactions get scored.

Frame it as: reduce steady-state cost without touching the burst-absorption guarantee, since the warm pool is the SLO insurance.

Options:

1. Right-size the warm pool. Measure actual peak concurrency per flash sale over 8 weeks. If real peak is 2,200 req/s rather than 3,000, the pool shrinks proportionally. Cheapest win, no model change.
2. Model efficiency. Quantize (FP16/INT8) or distill to a smaller model. If p99 stays under budget at higher per-node throughput, fewer nodes are needed. Validate on a recent labeled sample that fraud recall does not degrade beyond an agreed tolerance - this is a risk trade, not free.
3. Batch requests server-side. Micro-batching within a few milliseconds can raise GPU throughput substantially for small models. Adds a little latency; check it fits the 1.5 s queue deadline.
4. Spot/preemptible for burst replicas only. Burst capacity tolerates interruption because the warm pool covers the gap; steady-state replicas stay on committed capacity.
5. CPU inference for a subset. If a lightweight model variant meets latency for low-value transactions, route them there and reserve GPUs for high-value scoring.

Decision rule: rank by (savings × confidence) ÷ risk to SLO and model quality. Options 1 and 4 are low-risk and reversible; take them first. Option 2 changes model behavior and needs offline validation plus a canary with fraud-recall monitoring. Option 3 changes latency profile and needs load testing. Reject any option that removes the warm pool or the deterministic fallback, because those protect against the failure mode compliance cares about. Present a cost model showing savings versus expected dropped-transaction cost at each step.