Endpoint cost blowout on a recommender fleet
A retail team runs five models: three online recommenders (40 req/s steady, 300 req/s at the daily flash sale), one embedding model used by a separate search service at 2 req/s, and one ranking model used only by a nightly job that scores 50 million catalog rows. Every model currently has its own Vertex AI Endpoint with two GPU-backed replicas. Cloud Monitoring shows GPU utilization near 8% on the embedding endpoint and around 35% on the recommenders outside the sale window. Monthly endpoint spend is the largest line item on the team's bill. Latency SLO for the online recommenders is p95 under 150 ms; the nightly job has a 4-hour window and no latency requirement.
1. Restructure the serving topology to cut cost without breaking the SLO. What goes where?
Read the worked answer
Match the deployment mechanism to the traffic shape rather than defaulting everything to an endpoint.
- Nightly ranking job -> Batch prediction. 50M rows with a 4-hour window and no latency requirement is exactly what batch prediction is for. It spins up compute, scores, writes to BigQuery, and tears down. Paying for always-on GPU replicas to serve a job that runs once a day is the single largest waste on this bill.
- Embedding model -> co-host. At 2 req/s it does not need dedicated capacity. Deploy it to the same endpoint as a low-traffic recommender using a shared deployment resource pool so both models share the underlying machines. Watch for memory footprint: co-hosting fails if the combined model artifacts exceed the machine memory.
- Three online recommenders -> dedicated endpoint(s) with real autoscaling. These carry the SLO, so they keep their own capacity, but 35% utilization at steady state means replicas are oversized. Right-size the machine type first, then set min replicas to cover steady state and let autoscaling absorb the sale spike.
Measure per-model cost after the change; co-hosting saves money only if the shared pool is genuinely utilized.
2. How should autoscaling be configured for the online recommenders given the flash-sale spike?
Read the worked answer
Target utilization is the key number, and 35% steady-state utilization says the current shape is wrong. Set min replicas to the number that handles steady 40 req/s at your latency budget, and max replicas to what 300 req/s requires, computed from measured per-replica throughput rather than guessed.
Specifics that matter:
- Scale on the right signal. For GPU-backed models, CPU-based autoscaling often reacts late because the GPU saturates before CPU does. Enable DCGM metrics and scale on GPU utilization (or on concurrency) if the model is GPU-bound.
- Set target utilization around 60-70%, not 100%. Queuing at the replica drives p95 latency up sharply before utilization looks alarming.
- Warm-up is the real risk. New replicas take time to load model weights; if the flash sale ramps in minutes, reactive autoscaling will breach p95 before the new replicas are ready. Either schedule a min-replica bump ahead of known sale windows, or accept a higher min replica count during the event.
Validate with a load test at 300 req/s before the next sale, not during it.
3. A retrained recommender is ready. How do you release it, and what would make you roll back?
Read the worked answer
Canary with traffic split on the existing endpoint: deploy the new model version alongside the current one, shift 5% of traffic, and hold for a defined observation window that covers at least one full traffic cycle (a few hours, ideally including a peak).
Watch three things, decided before the release:
- p95 latency and error rate on the canary slice versus control.
- Prediction distribution shift - if the new model's score distribution differs sharply from the incumbent at similar inputs, that is a signal worth understanding before widening.
- A business proxy such as click-through on the canary slice, if it arrives fast enough to be actionable.
Promote to 50% then 100% only if the canary clears thresholds. Roll back by resetting traffic split to 0% for the new version - keep the old version deployed until the canary is fully promoted, because redeploying an old model under pressure is slower than shifting traffic away from a bad one.
Record the model version and split percentages in the release ticket so an incident review can reconstruct exactly what served traffic when.