When the Classifier Is Confident and Wrong
An operations team built an AI classifier that routes inbound support emails into three queues: Billing, Technical, and Cancellation. On a 500-email labeled test set it scored 91% accuracy, and the team shipped it with a rule that any email with model confidence below 0.70 goes to a human triage queue. In production, 8% of emails fall below 0.70. After three weeks, the Cancellation queue lead reports that several high-value customers were routed to Billing and only discovered the misroute when the customer called. Those emails had confidence scores of 0.88-0.95. The test set was sampled from the previous year and was 70% Billing, 22% Technical, 8% Cancellation. Production traffic is 45% Billing, 40% Technical, 15% Cancellation, and Cancellation emails are frequently short and terse ('cancel my account, effective immediately').
1. Explain why the confidence threshold did not catch the Cancellation misroutes, and why the 91% test accuracy is misleading for this deployment.
Read the worked answer
Two separate problems compound here.
The threshold is measuring the wrong thing. A confidence score reflects how strongly the model favors one class given its training. A short, terse 'cancel my account, effective immediately' shares vocabulary with billing disputes ('my account', 'effective', dates, amounts). The model can be very confident and still wrong, because confidence is not correctness - it is the model's internal certainty, which is highest exactly when the input superficially resembles a familiar pattern. A low-confidence cutoff catches ambiguity, not systematic bias.
The test set does not represent production. The test set was 70/22/8; production is 45/40/15. Accuracy on a Billing-heavy sample is dominated by Billing performance. If the model gets Billing right 97% of the time and Cancellation right 60% of the time, overall accuracy can still look like 91% while Cancellation - the class with the worst errors and the highest business cost - is badly served. The 8% Cancellation share in the test set means roughly 40 examples, far too few to trust a per-class estimate, and the sample is also a year old, so language has drifted.
The 91% number answers 'how often is this right on last year's mix?' It does not answer 'how often does this misroute a cancellation today?' Those are different questions and only the second one matters.
2. The team wants to keep the model but add safeguards. Propose a monitoring and routing design that would have caught these errors earlier, and state what you would measure.
Read the worked answer
Measure per-class, on current traffic, weighted by cost.
- Track precision and recall per queue, not overall accuracy. For Cancellation, recall is the metric that matters: a missed cancellation is expensive; a false Cancellation is cheap to correct.
- Build a confusion matrix from a weekly stratified sample of production emails (say 100, oversampling Cancellation), hand-labeled. This catches drift the old test set cannot.
- Track the distribution of predicted classes versus expected mix. If Cancellation predictions drop below the ~15% base rate, something is wrong.
Route by risk, not just by confidence.
- Add a second signal: keyword/pattern rules for high-stakes intents ('cancel', 'close my account', 'terminate') that force human review regardless of model confidence. Cheap, and it directly addresses the failure mode.
- Send low-confidence and high-value-account emails to human triage. Account tier is available and is a better risk proxy than confidence alone.
- For the Cancellation queue specifically, require human confirmation before any irreversible action (account closure, refund). Misroutes then become a delay, not a loss.
Close the loop. Log every human correction with the original email and the model's prediction. Review weekly. When a pattern repeats (terse cancellations misread as billing), add it to the rule layer immediately and to the next training set. Confidence thresholds are a blunt instrument; per-class recall, cost-weighted review, and a correction feedback loop are what actually catch this class of error.