The Support Triage Pilot That Looked Great in Week One
A regional insurer deploys a ticket-classification model to route inbound support emails to 14 queues. In the two-week pilot, 1,200 tickets were processed. The model's overall accuracy was 91%, so the steering committee wants to roll it out to all 40,000 monthly tickets next month. A junior analyst notes that 78% of pilot tickets were 'policy cancellation' messages, and the model assigns that label to 84% of all tickets it sees, including billing disputes and claims questions. Median handling time fell 12% during the pilot, but the pilot ran during a period when the contact center had 15% more staff than usual. Agents also quietly re-routed misclassified tickets without logging it, because the re-route button was buried in a submenu.
1. Should the committee approve the full rollout next month? Justify your recommendation using the evidence in the setup.
Read the worked answer
No - approve a controlled expansion, not full rollout.
Why the 91% is misleading. The pilot is heavily skewed: 78% of tickets are cancellations. A naive model that always predicts 'cancellation' would score roughly 78% accuracy on this sample. The model predicts cancellation 84% of the time, so it is barely beating that trivial baseline. Overall accuracy is the wrong metric for a 14-class problem with this class imbalance; per-class precision and recall (and a confusion matrix) are needed, especially for low-volume but high-stakes queues like claims.
Why the operational signal is confounded. The 12% handling-time drop coincides with 15% more staffing. You cannot attribute the gain to the model. Worse, agents are silently correcting errors off-system, so the logged data understates error rate - the true misclassification rate is unknown.
Recommendation. Hold full rollout. Fix instrumentation first (log every re-route), re-evaluate on stratified samples with per-class metrics, and run a staged expansion with a holdout cohort and a defined rollback trigger.
2. What single measurement change would most improve the committee's ability to make this decision, and why that one?
Read the worked answer
Make agent re-routes a first-class, logged event - ideally with the true label captured.
Reasoning. Right now the system's recorded performance is contaminated: the model's errors are being fixed by humans off the record. That means every downstream metric (accuracy, handling time, escalation rate) is measuring a human-machine team, not the model. Logging re-routes does three things at once:
- Produces a ground-truth label stream for real production traffic, enabling honest per-class precision/recall.
- Lets you measure the true error rate and whether errors cluster in specific queues (e.g., claims).
- Separates model effect from staffing effect, because you can compare handling time on corrected vs. uncorrected tickets.
It is also cheap and fast compared to retraining or a new data collection effort. Per-class metrics are important too, but they are only trustworthy once the label contamination is fixed.