Ninety-One Percent Against a Ninety-Six Percent Bar
A telecom operator receives 60,000 support tickets per quarter, each labelled into one of 42 categories by human agents. A prompt with category definitions and eight few-shot examples on a hosted chat model reaches 91% top-1 accuracy against a 96% bar required to auto-route without human review. Adding retrieval over 5,000 previously labelled tickets raised accuracy to 93% but pushed p95 latency from 2.1 s to 3.4 s and doubled token cost. The category taxonomy is revised every quarter: two or three categories are merged, renamed or added. The operator also wants a one-sentence justification in the agent's language, which may be English, German or Spanish. The team has 60,000 labelled examples, of which 12,000 are from the current taxonomy, and a dedicated AI cluster already used for another workload's inference.
1. Should the team fine-tune, keep improving the prompt, or extend the retrieval approach? Justify the split between learned behaviour and changing knowledge.
Read the worked answer
The 96% bar and a quarterly-changing taxonomy pull in opposite directions, so separate what is learned from data from what changes. Category definitions, names and merges are knowledge and belong in the prompt, where a taxonomy revision is a config edit rather than a retrain. Mapping messy ticket text onto a category is a task style the base model already handles at 91%; the missing five points are likely domain vocabulary, ticket-specific phrasing and the rare-category long tail, which is what supervised fine-tuning on the 12,000 current-taxonomy examples can learn. Fine-tuning on stale labels teaches retired categories, so filter to the current taxonomy and hold back a stratified test set. Retrieval over labelled tickets is really dynamic few-shot: it helps but costs latency and tokens on every call. A sound plan is to fine-tune first, keep definitions in the prompt, re-evaluate after each quarterly revision, and keep retrieval only for classes the confusion matrix shows are still weak.
2. How should the team evaluate the classifier and set a threshold for auto-routing?
Read the worked answer
Aggregate accuracy hides the decision. With 42 classes, some rare, a single number can conceal a category at 60% recall that auto-routes wrongly every day. Evaluate on a stratified test set from the current taxonomy and report per-class precision and recall, a confusion matrix, and calibration, because the real question is when to route automatically, not what the model scores overall. Set per-class thresholds from that matrix: high-confidence classes auto-route, low-confidence classes and historically confused pairs go to a human queue with a suggested label. For confidence, use whatever the endpoint exposes; if token log probabilities are available use them, and if not use agreement across a few samples or a second model as verifier, validating that the proxy actually correlates with correctness before trusting it. Re-run the evaluation on every taxonomy revision and monthly on fresh tickets to catch drift, and watch the human-override rate, which moves before accuracy does.
3. What are the deployment and cost implications of serving a fine-tuned model, given the existing dedicated AI cluster?
Read the worked answer
Fine-tuning and inference are separate capacity decisions. A fine-tuning job runs on a fine-tuning dedicated AI cluster, one job at a time, while serving a custom model requires hosting it on a hosting dedicated AI cluster; custom models are not served from the shared on-demand pool. The cost is therefore not the price of one fine-tune but a fine-tune plus a cluster billing per unit-hour for as long as the endpoint exists, whether or not tickets arrive. Model that against roughly 650 tickets per day, a light load, and ask whether the accuracy gain justifies continuous dedicated hosting versus scheduled batch scoring. Options worth pricing: run the fine-tuned model in batches rather than synchronously; keep the base model on-demand for the long tail and use the fine-tune only for confused classes; or, if reusing the existing cluster, confirm that a hosting cluster serves only one model at a time before assuming sharing is possible. Budget also for quarterly re-fine-tuning and evaluation, the recurring costs teams forget.