Routing Support Tickets Between a Classifier and a Generative Model
A support team receives about 4,200 tickets per day across 11 product lines. Leadership wants each ticket tagged with a product line and a severity from 1 to 4, then wants a draft reply written for the agent. A vendor demo shows a single large chat model doing all three jobs from one prompt. In a pilot on 900 historical tickets, the single-prompt approach tagged product line correctly 81% of the time and severity correctly 68% of the time, and the draft replies were judged helpful in 74% of cases. A separate small supervised classifier, trained on 6,000 labeled tickets, reached 96% product-line accuracy and 91% severity accuracy. The team has two weeks and one part-time data engineer. Late-arriving tickets must be tagged within 30 seconds; draft replies can take up to 5 minutes.
1. Should the team keep the single-prompt design or split the workload? Justify using the pilot numbers and the stated constraints.
Read the worked answer
Split the workload. The pilot shows the chat model is weakest exactly where precision matters most: severity at 68% means roughly one in three tickets is mis-prioritized, which directly affects response ordering. The supervised classifier, trained on 6,000 labels, is far stronger on both tagging tasks (96% and 91%).
Reasoning steps:
- Match each job to the capability that fits it. Tagging into a fixed set of labels is a classification problem with a known label space; it rewards labeled data and gives measurable accuracy. Writing a free-text reply is a generative task with no single correct answer, judged by helpfulness.
- Use the latency budget. The 30-second tagging requirement is easily met by a small classifier and is at risk with a large chat model under load; the 5-minute reply budget is generous enough for a generative call.
- Respect the resourcing limit. Two weeks and one part-time engineer favor reusing the already-trained classifier rather than building a new multi-task prompt pipeline and re-tuning it.
So: classifier for product line and severity, generative model only for the draft reply, with low-confidence classifier outputs routed to a human.
2. The team worries that a 96% accurate classifier is 'not AI enough' compared with the chat model. How should this concern be addressed?
Read the worked answer
The concern confuses sophistication with fitness. Both are AI; they solve different problem shapes.
- A classifier trained on labeled examples is a supervised learning model. It is the appropriate tool when the output space is small, fixed, and measurable, which is the case for 11 product lines and four severity levels.
- A generative chat model is appropriate when the output is open-ended text where many answers are acceptable.
Judgement criteria to present to stakeholders:
- Error cost. A severity error changes queue order and can breach service commitments; a mediocre draft reply is corrected by the agent before sending. The classifier's higher accuracy therefore matters more.
- Measurability. Classifier accuracy is a single number tracked against a held-out set. Draft helpfulness requires human review, which is slower and noisier to monitor.
- Cost and latency. Per-ticket classification is cheap and fast; per-ticket generation is heavier. Spending generation budget on tagging buys nothing.
Recommendation: keep the classifier, publish its accuracy and confidence distribution, and reserve the generative model for the drafting step where its flexibility is the actual benefit.