A latency SLO that collapses at the start of every flash sale
A retail recommendations team serves a ranking model on a Vertex AI endpoint. Steady traffic is 400 QPS; flash sales drive 15-minute bursts to 2,400 QPS. The endpoint runs a custom container on n1-standard-4 with min replica count 1 and max 3, autoscaling on CPU utilization at 60%. p95 latency is 110 ms at steady state but rises to 900 ms with 5xx errors at the start of each burst, and the team is paged roughly twice a week. Each request reads 40 features from a BigQuery table, adding about 20 ms and occasional multi-second outliers. The container loads a 1.2 GB embedding table from Cloud Storage at startup. 60% of traffic is a 'you may also like' email digest sent at 8 AM daily. The monthly budget is fixed and currently exceeded by 30%.
1. Explain the burst failures. Which single change buys the most, and what does the autoscaler configuration need to look like?
Read the worked answer
The endpoint is structurally unable to absorb a 6x step change. max replicas = 3 caps throughput regardless of demand; scaling on CPU at 60% reacts after queues have already formed; each new replica must pull 1.2 GB from Cloud Storage before it can serve, so scale-out latency is minutes while the burst ramp is seconds; and every request issues a BigQuery query, which is the wrong engine for point lookups - it adds slot contention and produces the multi-second tail.
Biggest single win: stop querying BigQuery per request. Precompute features into a low-latency store (Vertex AI Feature Store online serving or Bigtable) written by the same batch pipeline that trains the model. That removes the tail and shortens the critical path.
Autoscaler: set min replicas to cover steady state (roughly 400 QPS divided by measured per-replica throughput, not 1), raise max well above the 2,400 QPS requirement, and scale on a leading signal - concurrency or QPS per replica - rather than CPU, because CPU lags arrival rate. Bake the embedding table into the image or memory-map it from a regional bucket, and if startup cannot be reduced below a few seconds, keep pre-warmed headroom so new replicas are never cold.
Sequence matters: fix the data access first, then tune replicas, then re-measure. Tuning an autoscaler on top of a 20 ms-plus-tail lookup will just move the page.
2. The endpoint is 30% over budget while missing its SLO. Where do you cut cost without touching the interactive experience?
Read the worked answer
Start with the workload split, not the machine type. The 8 AM email digest is 60% of traffic and is entirely non-interactive: move it to Vertex AI batch prediction against a BigQuery source and table destination. Batch removes it from the endpoint, uses the same model artifact, and is dramatically cheaper per prediction. That single change reduces online traffic to roughly 160 QPS steady state, which changes the replica math and the budget picture at once.
Then: right-size. With 160 QPS steady state, choose the smallest machine type that meets p95 with headroom, and prefer more small replicas over fewer large ones for burst elasticity. Set min replicas to the real baseline rather than an inflated peak. Confirm GPU is not indicated - a ranking model of this shape typically does not need one, and if a GPU is in use, that is a large line item to justify or remove.
Do not cut cost by lowering the SLO or by letting the endpoint autoscale down to zero; cold starts on a step-traffic workload are exactly the failure you were paged for. If the budget still does not close, negotiate committed use on the baseline and report the burst capacity as a line item the business funds, because flash sales are the revenue event it exists for.
3. The team wants to ship a retrained ranker next week. Describe the rollout and what you would watch, given that the last two incidents were latency-related rather than quality-related.
Read the worked answer
Roll out in two dimensions at once, because the historical failure mode is operational.
Quality: register the new version in Vertex AI Model Registry and deploy to the same endpoint behind a traffic split - 5% for a defined window, comparing online metrics (click-through, add-to-cart) against the champion and checking that predicted score distributions have not shifted on a segment you care about.
Performance: the canary replica carries live traffic, so watch p95/p99 latency, error rate, and per-replica concurrency on the new version specifically, not endpoint-wide aggregates that average away the canary. If the new model has a larger embedding table or heavier preprocessing, the canary is where cold-start and memory behavior show up.
Promote in steps (5% to 25% to 100%) with an automatic rollback condition on error rate or p95 breach, not just on a human noticing. Keep the previous version deployed at 0% so rollback is an alias change rather than a redeploy.
Given two latency incidents, require a load test of the new version at 2,400 QPS before it takes any traffic. Quality regressions are caught by the canary in hours; capacity regressions are caught only if you look for them.