Canary decision for a larger ranking model against a hard latency budget

The NDCG number is not transferable. Three reasons:

1. Cache distribution. Production serves 40% of requests from cache, and cache hits are concentrated on head queries - the easy, high-volume, well-represented part of the distribution. The shadow run, with no cache, measured v7 on a much harder query mix than production actually sees. The +2.1% is an average over a distribution production will not experience. It could overstate the gain (tail queries improved) or understate it (head queries already served perfectly from cache). Either way the confidence interval on the production effect is unknown.
2. No feedback loop. Shadow scoring has no position bias, no click, no downstream conversion. Offline NDCG against logged relevance is a proxy for a proxy; it cannot see whether users respond differently to a reordered page.
3. Latency was measured at idle. 138 ms at 30% utilization is the good case. Queueing delay on a GPU is superlinear near saturation - going from 30% to 80% utilization will not scale p99 by 2.7x, it will scale it worse. The 120 ms budget is already breached by the lab number alone; the canary will run in a regime with less headroom than the lab.

On the canary itself: 5% for 48 hours is defensible in size but wrong in shape. It goes straight to 5% with no staged gate, it does not define what would trigger rollback, and 48 hours at 5% spends real money proving something a 1% stage could disprove in four hours.

Stages and duration (each stage gated, not time-gated alone): 1% for 4h -> 5% for 24h -> 25% for 24h -> 100%. Advance only if every gate passes at the current stage; hold if any gate is within 10% of its threshold.

Guardrails, measured on the canary bucket versus the v6 control bucket from the same traffic:

• Latency: p99 < 110 ms at every stage. Not 120 - you need headroom so that a traffic surprise does not breach the SLO before the gate notices. Also watch p999 < 250 ms and GPU queue depth.
• Errors: 5xx rate delta < 0.1 percentage point; timeout rate < 0.05%.
• Quality: live A/B NDCG@10 measured on real engagement, not offline relevance, with a 95% CI excluding zero improvement by the 25% stage. Also query abandonment and revenue per search as business guardrails.
• Tail queries specifically: NDCG on the bottom decile of query frequency, because that is where the shadow evidence is weakest and where the head-heavy cache is absent.
• Capacity: GPU memory utilization < 85% per replica, and replica count must allow one node failure without breaching latency.

Automatic rollback (no human in the loop): p99 > 120 ms sustained for 5 minutes; error delta > 0.25 pp for 10 minutes; any business guardrail down more than 1% versus control for 30 minutes. Rollback is a traffic-weight change back to 0%, with v6 replicas kept warm and un-scaled-down for the entire rollout - never rely on scaling v6 back up during an incident.

Memory first, because it is a hard wall. Four v6 replicas at 78% utilization means each replica currently uses 78% of the GPU's memory. A v7 replica needs 1.6x that v6 usage. If v6 usage per replica is M and the card holds C, then 0.78C = M, so v7 needs 1.6 x 0.78C = 1.25C - it does not fit on the same card at all, at any replica count. This is the constraint the proposal ignores. Options, in order of preference: quantize to INT8 (typically cuts weights roughly 2-4x, which brings 1.25C well under C, but re-validate NDCG because quantization changes scores); or move to a larger card with proportionally more memory; or shard/offload, which adds latency and is a bad fit for a 120 ms budget.

Compute. At steady state 12,000 QPS with a 40% cache hit means about 7,200 QPS reaches the model. If one replica sustains roughly 1/0.138 s of serialized capacity at p99, target utilization of 60-70% (not 30%) for headroom. Replicas = 7,200 / (1/0.138 x 0.65) - approximately 7,200 x 0.138 / 0.65, so about 1,530... which is the wrong unit; the correct form is 7,200 x 0.138 s of work per second divided by 0.65, giving roughly 1,530 GPU-seconds per second, i.e. about 1,530 replicas if one replica took 0.138 s per request. That number is absurd, which is the point: the 138 ms figure must be a batched-server p99, not per-request serialized time. Confirm what 42 to 120 requests per second per replica the lab actually measured before committing to any pool size - the capacity plan is only as good as that measurement, and the 30%-utilization p99 cannot be converted into throughput.