Support Ticket Classifier with Noisy Outage Labels
A SaaS support organization has 200,000 historical tickets labeled into 14 routing categories by an offshore vendor. A quality audit of 1,000 tickets finds 9% labels disagree with an internal taxonomy expert; disagreement is 31% for the outage category and 4% for billing. A text classifier trained on the full set achieves macro-F1 0.63, with billing F1 0.81 and outage F1 0.34. The routing team handles 25,000 tickets per month and wants to auto-route only when confidence is high, sending the rest to a human queue. Misrouting an outage ticket delays a Sev-1 response and costs far more than misrouting a billing ticket. The model owner proposes collecting 5,000 new clean labels and using active learning. The budget covers only 3,000 expert labels this quarter.
1. How would you determine whether the outage F1 of 0.34 is a model problem or a label problem?
Read the worked answer
Build a clean, adjudicated test set before judging the model. Stratify a sample by predicted and true labels, have two internal taxonomy experts label independently, measure Cohen's kappa, and resolve disagreements to a gold label. Evaluate the current model on that gold set. If outage F1 rises substantially, the 0.34 is mostly label noise; if it stays low, the model or features are weak. Also inspect the vendor's confusion for outage: are outage tickets labeled as bug or account? A 31% disagreement rate is high enough that the training labels for outage are not trustworthy. Check whether outage tickets are rare and whether the model's errors concentrate on short tickets with no diagnostic text. Only after separating label noise from model error should you choose remediation.
2. Recommend a labeling and training strategy under the 3,000-label budget.
Read the worked answer
Spend the 3,000 labels where they reduce uncertainty. Use active learning with diversity: select tickets near the decision boundary, tickets predicted as outage with low confidence, and a random control sample. Stratify by class and by ticket length. Have two annotators label a subset and adjudicate disagreements; use the rest for single-expert labels with audit. Keep a held-out gold set of at least 300 tickets untouched for final evaluation.
Train with label-noise-robust methods: co-teaching, confident learning, or simply down-weighting suspect labels after audit. Do not only add class weights to outage; that amplifies noisy labels. Consider a two-stage router: first detect outage versus non-outage with a high-recall model, then classify the remaining categories. The 3,000 budget is enough for a reliable outage-focused audit if sampling is targeted, not random.
3. How should the system decide when to auto-route versus escalate?
Read the worked answer
Use class-specific thresholds and an abstain option. For outage, set a high-precision threshold and route anything below it to a human queue, because a missed outage is a Sev-1 cost. For billing, a lower threshold is acceptable because the cost of misrouting is small. Estimate the cost matrix with the routing team: cost of human review, cost of delayed Sev-1, and cost of a wrong route. Choose thresholds that minimize expected cost subject to a maximum auto-route rate.
Calibrate probabilities first; otherwise confidence scores are not comparable. Monitor auto-route coverage and precision by class weekly. Add a feedback loop so human corrections become labeled data. Never auto-route an outage prediction with low confidence just to reduce queue volume.