Intent Router Under Holiday Drift and a 45 ms Latency Budget
ShopStream's intent router classifies 12 customer intents at 3,000 requests per minute. The p95 latency budget is 45 ms on CPU because the router sits in front of a chat queue. A DistilBERT model achieves macro-F1 0.84 but p95 latency 340 ms on the current CPU fleet; on GPU it is 90 ms, but GPU capacity would add 38% to monthly serving cost. During a holiday campaign, 'where is my order' traffic tripled and new slang such as 'WISMO' and 'shipment stuck' appeared. Weekly macro-F1 fell to 0.69, and the confidence distribution shifted; 22% of requests now fall below the 0.55 abstain threshold. The team retrains quarterly. A proposal is to serve the transformer only when a fast linear model's confidence is below 0.7; otherwise route directly.
1. Evaluate the cascade proposal against quantizing the transformer. Recommend an architecture that respects the latency budget and explain the key trade-offs.
Read the worked answer
The cascade is attractive because most traffic is easy. Measure the linear model's coverage at threshold 0.7: if it confidently handles 70% at about 5 ms, only 30% of traffic goes to the transformer. At 3,000 requests per minute, that is 900 requests per minute to the transformer; at 90 ms on GPU with batching this may fit, but the 38% cost estimate assumed all traffic on GPU, so recompute rather than reuse it. The real risk is the 22% of requests below 0.55: the cascade's low-confidence branch will be overloaded during drift. Use a three-tier design: linear high-confidence routes directly; mid-confidence goes to a distilled or quantized transformer; low-confidence or out-of-distribution goes to a human queue. Quantize the transformer with ONNX Runtime INT8 and measure p95 on the actual CPU fleet; if it reaches 110–130 ms, it still misses 45 ms and cannot be the first tier. A smaller distilled model may reach 25–35 ms with some F1 loss. Compare end-to-end p95, macro-F1, cost per 1,000 requests, and abstain rate, not model F1 alone. Canary the chosen design and keep the linear fallback.
2. Propose a production monitoring plan that would have detected the holiday drift earlier. Specify signals, thresholds, and slicing.
Read the worked answer
Monitor four layers. Input: per-intent query volume, new-token and out-of-vocabulary rate, language mix, and population stability index on embeddings or TF-IDF; alert when tokens such as 'WISMO' spike. Prediction: confidence distribution, abstain rate, entropy, and per-intent share; the rise from the pre-campaign abstain baseline is an early warning. Outcome: delayed human-agent disposition and CSAT by intent; compute weekly macro-F1 when labels arrive. System: p95 latency, error rate, and queue depth. Set thresholds from pre-campaign baselines, for example abstain rate above 1.5× baseline for 24 hours, PSI above 0.2, or per-intent F1 drop above 0.05 triggers review. Use a shadow model on 5% of traffic to compare candidate retrains without user impact. Dashboards should slice by intent and channel because aggregate F1 masked the 'where is my order' surge. Include a data-quality check for label lag and duplicate tickets.
3. The team retrains quarterly. Recommend a retraining, promotion, and rollback policy that handles a fast campaign shock.
Read the worked answer
Quarterly retraining is too slow. Move to trigger-based retraining: when abstain rate or PSI crosses threshold, pull the last 8 weeks of labeled dispositions, oversample new intents and slang, and fine-tune from the current model rather than from scratch. Keep a frozen evaluation set with a campaign slice; require macro-F1 improvement of at least 0.02 overall and no intent drop greater than 0.03 before promotion. Use a 5% canary for 48 hours, then 25%, then 100%, with automatic rollback if p95 latency exceeds 45 ms, error rate doubles, or abstain rate rises. Maintain a rule-based fallback for 'where is my order' and other top intents so the system degrades gracefully. Version preprocessing and model together: adding slang tokens can invalidate the linear model's feature indices. Log model version, preprocessing version, threshold, and fallback reason for every prediction so an audit can reconstruct why a request was routed.