Nine ticket categories, one tuning budget, and a new plan nobody labelled
A telecom's data science team wants to route incoming support tickets into nine categories. A prompt-only baseline using a Granite instruct model reaches 0.71 macro F1 on a held-out set of 300 tickets. Errors concentrate between billing-dispute and payment-failure, and between two network-outage classes. Each ticket averages 900 tokens, and a few contain pasted device logs of 4,000 tokens. The team has budget for one tuning run and one deployment. A vendor proposes fine-tuning a 70B model on 900 training tickets; an internal engineer argues that a retrieval layer over the last six months of resolved tickets would be cheaper. After go-live, a new unlimited-data plan launches and ticket volume for one class triples, with wording the model has never seen. The manager needs a defensible plan for model choice, tuning configuration, deployment type, and post-launch monitoring, and must explain to audit why the approach is proportionate.
1. Tuning, retrieval, or prompt engineering first - how do you decide, and what do you recommend given the error pattern?
Read the worked answer
The error pattern is the deciding evidence. Confusion is concentrated between sibling classes - billing-dispute versus payment-failure, outage type A versus B - which is a decision-boundary problem, not a knowledge-recall problem. Retrieval helps when the model lacks facts; it will not teach it where the line sits between two similar labels, and pulling in six months of resolved tickets mostly injects label noise and long, expensive context.
So: first harden the prompt - write explicit class definitions, include the disambiguation rule for the two confused pairs, and add a handful of few-shot examples per class in a fixed order. That costs a day and often moves macro F1 by several points.
If that plateaus, tune a mid-size instruct model rather than a 70B. Reasons: 900 examples is small for a large model and risks overfitting, the 70B costs far more per inference at ticket volume, and the task's output space is narrow. A smaller tuned model with a constrained output format is the proportionate choice, and it is easier to justify to audit as a controlled change on a defined dataset.
Separately, address the 4,000-token tickets: head-and-tail truncation plus a one-line summary of the middle, or route them to a slow path. Tuning does not fix missing context. Keep the prompt baseline deployed as a fallback so a tuning regression is a switch, not an outage.
2. What tuning configuration and evaluation discipline would you apply, and what would make you reject the tuned model?
Read the worked answer
Configuration decisions follow from the data shape and the failure modes you must protect.
- Input length. Set max sequence length above the realistic ticket length after truncation, not at a default. If the setting silently clips at 512 tokens, you have trained on the first half of every ticket and will wonder why outage tickets still fail.
- Epochs. Start modest - a handful - and watch validation loss; with 900 examples, more epochs buys memorization, not generalization. Early stopping on validation macro F1.
- Effective batch size and learning rate. Use gradient accumulation to reach a sane effective batch on limited hardware, and a conservative learning rate; this is a classification-shaped problem, not a full pretraining run.
- Parameter-efficient tuning. Low-rank adapters keep the run cheap and fast, which matters when you have exactly one tuning attempt.
- Class balance. Examine the training set per class; if one class has 40 examples, augment or merge before blaming the model.
Evaluation discipline: the same held-out 300 tickets, the same metric (macro F1, since per-class performance matters more than aggregate accuracy), plus the confusion matrix compared class by class against the baseline. Report per-class precision, because a routing system that over-assigns to one queue creates a staffing problem even at similar overall F1.
Reject the tuned model if macro F1 does not improve beyond noise, if any class regresses materially, or if it becomes brittle on the truncated long tickets. Document the comparison so audit sees a controlled experiment rather than a preference.
3. Online or batch deployment, and what monitoring would have caught the post-launch shift?
Read the worked answer
Deployment type follows the latency requirement, not fashion. Tickets arrive continuously and routing must happen in seconds to be useful, so this is an online deployment in a deployment space, with the prompt baseline available as a second deployment for rollback. Batch would only make sense for nightly re-scoring of historical tickets, which is a reporting job, not routing.
Monitoring should cover three layers, because model quality labels arrive late:
- Input drift. Track the distribution of ticket text embeddings and the rate of vocabulary the model never saw in training. The new plan's wording would surface here within days, before any quality complaint.
- Prediction drift. Track the share of predictions per class. One class tripling in volume is visible immediately in the prediction mix even without ground truth, and should trigger review rather than silent acceptance.
- Delayed quality. Agents override routing constantly; capture those overrides as labels and compute rolling accuracy and per-class F1 weekly. Set a threshold, for example a five-point drop in macro F1 or a doubling of override rate, as the retraining trigger.
Governance-wise, register the model in the inventory with its training data reference and evaluation results, and evaluate fairness across customer segments - routing errors that concentrate on one segment are a compliance issue, not just an accuracy issue. Keep prediction logs (with PII minimisation) so the next evaluation does not require a new data collection exercise.