Recommendation Serving at 340 ms p99

The 15 MB dense tower is not the problem; the 1.2 GB embedding table is. Three effects compound.

1. Memory. A 1.2 GB float32 table, resident per GPU process plus graph and workspace overhead, at 98% utilisation means one extra replica or one larger batch triggers the OOM. At 260 RPS during the flash sale the service was already past capacity.
2. Transfer. With the lookup in Python and the table copied per request, you pay host-to-device transfer and interpreter overhead on every call. Since the tower is tiny, nearly all latency is lookup, transfer and kernel-launch overhead - not matmul.
3. No batching. Per-request kernel launches at 200 RPS leave the GPU idle between calls; launch overhead dominates.

Ranked fixes: first, move the embedding gather into the serving graph and batch it so the table stays resident and one transfer serves N requests. Second, enable dynamic batching with a short window. Third, quantise the table (float32 to int8 is roughly a 4x cut, to ~300 MB) or shard it if the vocabulary keeps growing. Architecture change - retraining with a smaller dimension - goes last, because it costs a full retrain plus re-evaluation of ranking quality and may not be needed once the gather is fixed.

A 50 ms window is too large for this SLO, and the arithmetic shows why. Dynamic batching adds queueing delay equal to the time a request waits for the window to close. A request arriving just after a batch is dispatched waits nearly the full 50 ms, then pays compute and network. If a batched forward pass plus serialisation is around 20–30 ms, worst-case server time is 70–80 ms before any network or client cost - and the SLO is p99, so the worst case is precisely what you must budget, not the average.

Against a 300 ms mobile client timeout and a 120 ms p99 target with current p99 at 340 ms, this does not leave room to absorb variance, retries or a flash-sale spike.

The window is also solving the wrong problem. The gain from batching comes from amortising per-request launch and transfer overhead, and that is largely captured by a batch of 8 or 16. A 5–15 ms window with a maximum batch size achieves most of the throughput benefit at a fraction of the added latency. Under real load, queues form naturally and batching happens without any timer at all - the window mainly helps at low traffic, where latency matters least.

Recommendation: set the window to 5–10 ms, cap batch size, and measure the latency CDF at 50, 150 and 260 RPS rather than relying on a single average.

Before traffic. Shadow 100% of requests to the new stack for 24–48 hours without serving its output, comparing the latency distribution, error rate and top-10 overlap with the current system. Then a 5% canary, ramping only on clean signals.

Automatic rollback triggers, evaluated over a 5-minute window:

• p99 latency above 150 ms. The SLO is 120 ms; leaving only 30 ms of headroom on a 300 ms mobile timeout is already thin.
• Error rate above 0.5%, or any OOM or crashloop in the canary pool.
• GPU memory above 85% - the leading indicator that preceded the flash-sale OOM.
• Business guardrail: CTR or conversion down more than 2% relative with adequate sample. Pre-compute the sample size or the canary will run too long before anyone can judge it.

Rollback must be fast and rehearsed. Thirty minutes to fall back is not acceptable at this traffic. Pre-generate fallback lists hourly instead of nightly, keep the batch recommender warm and deployable, and execute one rollback during the canary so the path is proven before an incident.

Capacity. Plan explicitly for the flash sale: provision headroom for 2x peak and add a load-shedding rule so the service degrades to a cached popular-items list rather than OOMing. The 260 RPS spike should be a scheduled capacity test, not a surprise.