Endpoint Autoscaling and Deployment Safety
A recommendation model is served by a SageMaker real-time endpoint with 3 ml.c5.2xlarge instances. Traffic is diurnal: ~50 req/sec at night and ~900 req/sec during a 2-hour evening peak, with p99 latency target of 150 ms. The team enabled target-tracking autoscaling on InvocationsPerInstance with target 400 and a 300-second scale-out cooldown. During the peak, latency exceeded 2 seconds for 15 minutes and some requests returned 429s. Model load takes ~4 minutes per instance. The model artifacts are 1.8 GB and pulled from S3 on each scale-out. A new model version is ready and the team wants to deploy it without downtime.
1. Explain why autoscaling failed to hold latency and propose concrete configuration changes.
Read the worked answer
Three compounding causes. First, the target is too high: 400 req/sec per instance on c5.2xlarge for a model needing sub-150 ms p99 leaves no headroom, and the per-instance capacity is itself a guess. Second, the 300-second cooldown plus ~4 minutes of model load means a new instance is useful only ~9 minutes after the trigger, while the ramp to 900 req/sec happens faster than that. Third, the 1.8 GB artifact pull from S3 on each launch adds to cold-start time, so scale-out lags demand and the endpoint saturates, producing 429s.
Changes: lower the target to ~150–200 req/sec per instance so scaling starts earlier, and set a shorter scale-out cooldown (e.g., 60s) with a longer scale-in cooldown to avoid flapping. Pre-warm capacity before the known evening peak with a scheduled scaling action, since the diurnal pattern is predictable - this is the highest-value fix. Reduce cold-start by baking the model into a container image or using a smaller/faster-loading artifact, and consider a larger instance type so fewer instances are needed at peak. Set a minimum capacity that covers the night baseline so scale-in does not fight the next ramp. Add CloudWatch alarms on p99 and 429 rate as the real SLO signal, not just invocation count.
2. Design a zero-downtime rollout for the new model version and describe the rollback trigger.
Read the worked answer
Use a SageMaker deployment with traffic shifting rather than replacing the endpoint. Create the new model and endpoint config, then update the endpoint with a canary or linear traffic shift: send 10% of traffic to the new version, hold for a defined bake period, then step to 100%. During the bake, compare the new version's p99 latency, error rate, and a business metric (e.g., click-through or prediction distribution drift) against the incumbent, using the same CloudWatch metrics split by variant.
Rollback trigger: automatic rollback if error rate exceeds baseline by a set margin, p99 exceeds the 150 ms target, or the model output distribution shifts beyond a threshold, within the bake window. SageMaker supports automatic rollback on CloudWatch alarm during deployment, which removes the human latency from the decision.
Capacity caveat: during the shift both variants run, so the endpoint temporarily needs capacity for both. Pre-scale to cover the evening peak plus the canary share, or schedule the rollout outside the peak window. Also ensure the new artifact is pre-loaded (baked image or warmed instances) so the canary instances do not cold-start and distort the latency comparison - otherwise you may roll back a good model because of load time, not model quality.