Email Triage Model Confidence and Human Review
A pension provider receives 48,000 member emails per month. A Pega Process AI text categorization model routes each email to one of 14 work queues. On a held-out test set, top-1 accuracy is 71%, and top-2 accuracy is 89%. The current threshold sends an email to a queue only when the top prediction score is at least 0.60; otherwise it goes to a general queue. The general queue has a 3-day backlog. Operations wants to reduce misroutes, which cost about $18 each, while the general queue costs $6 per email and adds delay. A data scientist proposes lowering the threshold to 0.45 to reduce the general queue. Another proposes a human review step for scores between 0.45 and 0.70. The model was trained on 18 months of data and has not been retrained in 9 months.
1. Evaluate the two proposals using a cost-sensitive decision framework.
Read the worked answer
Use the break-even misroute rate. A general-queue email costs $6; a misroute costs $18. Lowering the threshold is worthwhile only if the incremental emails routed directly have a misroute rate below $6/$18 = 33%. If the 0.45–0.60 band has a higher misroute rate, lowering the cutoff saves queue cost but increases total cost. For the human-review proposal, compare the review cost plus any delay cost against the avoided misroute cost. Because top-2 accuracy is 89%, a reviewer presented with the top two queues can resolve many mid-confidence emails cheaply. Recommend: keep 0.60 as the auto-route threshold or lower it only after measuring the band's actual misroute rate; send 0.45–0.70 to human review with top-2 suggestions; send below 0.45 to the general queue. This uses the model's ranking rather than forcing a single cutoff.
2. What monitoring and retraining plan would you recommend?
Read the worked answer
Monitor input drift, prediction distribution, queue volumes, human override rate, and confirmed misroute rate. The model is 9 months old, so drift is plausible. Set alerts when weekly queue distribution shifts by more than a defined percentage or when reviewer overrides exceed a baseline. Capture reviewer corrections as labeled data, because they are the most relevant recent labels. Retrain on a rolling window that includes recent labeled corrections, but hold out a time-based test set to avoid leakage. Use champion-challenger: keep the current model live while a challenger is evaluated on live traffic or a shadow mode. Do not retrain only on routing outcomes, because those outcomes reflect the current model's choices. Include a random sample of reviewed emails to measure true accuracy.
3. How should low-confidence emails be designed in the case lifecycle?
Read the worked answer
Create a review assignment rather than dropping the email into the general queue. The assignment should show the email text, top two predicted queues with scores, and a one-click confirm or reassign action. Route it to a review pool with a short SLA, for example 4 business hours, and escalate if the pool backlog grows. If no reviewer is available, fall back to the general queue with a flag. Capture the reviewer's choice and the final outcome on the case so the label is available for retraining. Do not make the review step a child case unless the work needs separate ownership; a single assignment keeps cycle time low. Measure review volume, override rate, and time-to-route. This design reduces misroutes without adding a second backlog and turns human effort into training data.