AWS Machine Learning Specialty (AWS): A 250 ms SLO, a Three-Week Traffic Surge, and a Monthly Release Train

Why it failed: CPU is the wrong signal and a lagging one.

• The instances were memory-saturated, not CPU-bound, so the policy never triggered a scale-out even as requests queued and timed out.
• Scaling reacts after latency degrades. With a 25-second model load, every new instance is useless for the first half-minute of a spike - precisely the window where the queue explodes.
• Scale-in returned capacity too eagerly for a bursty load.

Changes:

1. Target a load metric, not CPU. Use InvocationsPerInstance target tracking, or a step-scaling policy on a CloudWatch alarm for p99 ModelLatency. Either way the signal tracks demand rather than one saturated resource.
2. Asymmetric cooldowns. Short scale-out cooldown (around 60 s), long scale-in cooldown (600 s or more) so capacity is not shed between bursts.
3. Scheduled scaling beats reactive scaling for a known event. Create a scheduled action that raises minimum capacity on day 21 and keeps it for the three weeks; do not fight autoscaling lag against a predictable ramp.
4. Fix the per-instance load. Use Inference Recommender to benchmark alternative instance types and batch sizes; if the model is memory-bound, a larger instance holds more concurrent requests and load-once cost is amortised. Pre-warm new instances with a synthetic request after deployment so the 25-second load is not counted against customer traffic.
5. Check the edges: ALB idle timeout above the endpoint timeout, client retries with backoff and jitter, and a circuit breaker so a retry storm does not compound the overload.

The incident was a detection and reversibility problem, not a modelling problem.

Reversibility:

• Register every model in SageMaker Model Registry with an approval status; only Approved versions can be deployed.
• Keep the previous endpoint configuration and model artifact intact. Rollback then becomes a single UpdateEndpoint with the prior config - minutes, not six hours - rather than a rebuild.

Detection:

• Deploy with deployment guardrails: a blue/green update with canary traffic (e.g. 10% for 10 minutes, then 50%, then 100%) and a baked period.
• Attach CloudWatch alarms on Invocation5XXErrors, ModelLatency p99, and a business metric (score distribution shift or approval rate) that trigger automatic rollback when breached.
• Run a smoke-test payload set against the new variant before any customer traffic is shifted, comparing outputs to the incumbent for a sample of known cases.

Before that:

• Optionally run the candidate as a shadow variant receiving mirrored traffic with no response obligation, and diff its predictions against production for a day. Shadowing is the only way to see behaviour on live inputs without risking customers.

Automate the whole path in CodePipeline so it is identical every month, and record model lineage so a bad release can be traced to the exact training job and dataset.

Layer three monitors, because the outcome signal is too slow to protect you.

1. Data capture. Enable DataCaptureConfig on the endpoint, sampling a percentage of requests and responses to S3. Without captured inputs there is nothing to monitor.
2. Data quality monitor. Build a baseline from the training data with Model Monitor's statistics and constraints, schedule hourly, and alarm on violations of the input schema and on feature distribution drift (compare against the baseline with a distance metric such as PSI). This is your early warning: an upstream pipeline change shows up here within an hour, not in 45 days.
3. Model quality monitor. Configure it against the ground-truth labels when they land. Because of the 45-day lag, also track proxy signals in the interim: the distribution of predicted scores, the approval rate per segment, and the correlation between the score and any available short-horizon outcomes (delinquency flags, manual review overrides). A sudden shift in score distribution is a leading indicator that the 45-day label will confirm later.

Wire alarms to two different destinations: 5xx and latency alarms to the on-call rotation with auto-rollback authority; drift alarms to a ticket that starts the retraining pipeline. Also run a bias monitor on protected attributes once labels arrive, since a drifting model can degrade fairness before it degrades accuracy.