Contact Center Triage: Separating Model Limits from Engineering Mistakes
A regional utility's contact center handles 42,000 support tickets a month. A two-person innovation team built a prototype on a large Gemini Pro model: every request pastes the full 60-page billing policy manual into the prompt, sets temperature to 1.0, and asks for a ticket category plus a drafted reply. Offline accuracy on category assignment is 71%, p95 latency is 9 seconds, and cost is $0.14 per ticket. The contact center director needs p95 under 2.5 seconds and under $0.01 per ticket before a January rollout. Reviewers also flagged drafted replies that cite policy clauses not present anywhere in the manual. An executive has read that fine-tuning on company data 'fixes hallucinations' and has asked the team to train a model from scratch.
1. Classify each observed problem as either an inherent property of large language models, a configuration choice, or an architecture choice. Justify each classification.
Read the worked answer
Invented policy clauses - inherent, but manageable. Hallucination is a property of probabilistic next-token generation; the model was asked to produce facts from its weights while the source text sat unused in the prompt. Temperature does not fix this: at temperature 0 the model picks the highest-probability token, which is why replies became repetitive but not more accurate. The fix is grounding - retrieve the relevant manual passages, require the answer to quote them, and permit 'not found'.
9s p95 and $0.14/ticket - architecture. ~60 pages are re-sent on every call, so input tokens dominate both latency and cost. Retrieval of 3-5 chunks, or context caching for a static preamble, changes the economics without touching model quality.
71% category accuracy - task fit. Category assignment is a narrow classification task; a smaller, faster model with a handful of examples and a structured output schema is usually sufficient.
Training from scratch - wrong lever. Pre-training buys world knowledge and general language ability, neither of which is the gap here, and it does not eliminate hallucination.
2. Recommend a target architecture and model strategy that plausibly meets p95 < 2.5s and < $0.01/ticket, and explain why pre-training from scratch is the wrong response to the hallucination complaint.
Read the worked answer
Split the workload instead of forcing one model to do both.
- Classification path: a small, low-latency model (Gemini Flash-Lite class) with 5-8 few-shot examples and a response schema that constrains output to the allowed categories. If labelled tickets exist, adapter tuning or distillation from the large model onto the small one typically recovers most of the large model's quality at a fraction of the cost.
- Reply-drafting path: retrieval over the manual (Vertex AI Search or a vector index), top-k around 4, answers citing retrieved passages, and an explicit instruction to say the manual does not cover the case rather than improvise. Context caching for the static instruction block cuts repeat input cost.
- Deterministic pre-routing for obvious categories (outage, billing cycle) removes volume before any model call.
Why not pre-train from scratch: you would spend millions to teach general language, not your billing policy; you lack the corpus; and the resulting model still generates plausible text. The gap is retrieval and evaluation, not world knowledge. Fine-tuning can shape format and tone - it cannot guarantee facts.
3. How would you prove the new design is better before the January cutover, and what would you measure?
Read the worked answer
Build a frozen evaluation set first, because without it every later decision is anecdote.
- Dataset: 300-500 real tickets, stratified by category and by channel, labelled by two reviewers with disagreements adjudicated; plus ~100 adversarial cases (out-of-scope requests, questions the manual genuinely does not answer, prompts inviting invented clauses).
- Metrics: category exact match; groundedness (does each citation actually support the claim - an autorater plus human spot-check); correct-refusal rate on out-of-scope; malformed output rate; p95 latency; cost per ticket.
- Procedure: shadow-run the new pipeline on live traffic alongside the prototype, compare against the baseline on the same inputs, and gate cutover on agreed thresholds (for example, category accuracy above 85%, groundedness above 95%, zero tolerated fabricated citations).
- Post-launch: weekly sampled human review plus drift monitoring on input distribution, since tax season and tariff changes will shift ticket mix.
Report a single scorecard to the director; do not let latency, cost, and quality be negotiated separately.