Blue/Green Rollout of a Recommender Endpoint
A retail team runs a product recommender behind a SageMaker real-time endpoint serving 4,000 requests per second at peak. A new model version improves offline NDCG by 3 points but changes the response schema: it adds two fields and renames one. Downstream clients include a mobile app that cannot be updated for three weeks and a web frontend that can ship in two days. The team wants to shift traffic gradually, detect regressions automatically, and roll back in under a minute. They currently deploy by updating the endpoint in place, which caused a 20-minute outage last quarter.
1. Design the deployment topology and traffic-shifting plan, accounting for the incompatible mobile client.
Read the worked answer
Topology. Move to a multi-variant endpoint (production variant plus a new variant) or, better, two separate endpoints behind an application-layer router. Separate endpoints give independent scaling and let the old version keep serving while the new one warms. In-place updates are the root cause of the prior outage; they replace capacity before new capacity is healthy.
Schema compatibility. The renamed field is a breaking change for the mobile app. Two options: (a) have the new model's serving container emit a backward-compatible response - include the old field name as an alias alongside the new one and add the two new fields - so both clients work against either version; or (b) route by client: mobile traffic pinned to the old endpoint until its release, web traffic shifted to the new one. Option (a) is preferable because it decouples deployment from client release schedules and avoids maintaining two live models longer than necessary.
Traffic shift. Canary in stages - 1 percent, 5 percent, 20 percent, 50 percent, 100 percent - holding at each stage long enough to collect statistically meaningful error and latency samples. Use weighted routing rather than all-at-once.
Rollback. Because the old endpoint stays warm, rollback is a routing weight change, achievable in seconds. Define automatic rollback triggers on error rate, p99 latency, and business metrics, and pre-authorize the rollback so no human approval delays it.
2. What should the automated guardrails measure, and how should the team avoid false alarms during the canary?
Read the worked answer
Measure three layers.
Infrastructure: invocation errors, model latency p50/p99, throttles, and instance health per variant. These catch capacity and container problems first.
Prediction quality proxies: for a recommender without immediate labels, track online proxies - click-through rate, add-to-cart rate, catalog coverage, and score distribution shift versus the incumbent. A sudden change in the distribution of recommended item categories often signals a feature or preprocessing bug before revenue moves.
Business outcome: conversion and revenue per session, compared against the incumbent over the same period.
Avoiding false alarms. Compare the canary against the incumbent concurrently rather than against last week's numbers - seasonality and promotions otherwise dominate the signal. Use sequential testing or a minimum sample size per stage so small canary slices do not trigger on noise; a 1 percent canary at 4,000 rps still needs enough conversions before a revenue metric is meaningful. Set thresholds on relative change with confidence bounds, not absolute values. Finally, separate hard failures (errors, latency) that roll back immediately from soft signals (CTR dip) that pause the rollout for human review - conflating them causes either reckless rollback or ignored warnings.