The Support Triage Classifier That Fails on Weekends
A SaaS company deploys a ticket-routing classifier trained on 18 months of historical support tickets. Overall validation accuracy is 94%, so the team ships it. Two weeks later, the operations lead reports that urgent billing disputes opened on Saturday and Sunday are frequently routed to the low-priority queue, delaying responses by up to 40 hours. Investigation shows that weekend tickets are only 6% of the training data, and that agents historically resolved weekend tickets faster (median 3 hours vs. 11 hours on weekdays), so the label 'resolved quickly' is overrepresented for weekend rows. The model uses features including ticket text, customer tier, hour-of-day, and day-of-week. The team proposes simply retraining on more recent data.
1. Diagnose why overall accuracy masked this failure, and identify the specific data and modeling problems at play.
Read the worked answer
Overall accuracy is a marginal metric: with weekend tickets at ~6% of rows, a model can be near-perfect on weekday rows and still score 94% overall while performing poorly on the weekend slice. The relevant diagnostic is slice-level performance (precision/recall on weekend-urgent-billing), not aggregate accuracy.
Two distinct problems compound:
- Distribution shift within the feature space - hour-of-day and day-of-week have very different conditional relationships on weekends, and there are few examples to learn them from.
- Label leakage / proxy-label contamination - the target 'resolved quickly' is partly a function of staffing, not urgency. Weekend agents resolve things faster, so the label encodes an operational artifact. The model learns 'weekend → fast resolution → low priority,' which is exactly backwards for the business goal.
Retraining on recent data does not fix either issue: it may even worsen the label contamination if recent staffing patterns persist. The fix is to redefine the label around true urgency (e.g., dispute amount, SLA breach risk, escalation history) and evaluate on the weekend-urgent slice specifically.
2. Propose a concrete remediation plan covering data, labeling, evaluation, and deployment safeguards, and state how you would measure success.
Read the worked answer
Data: Stratify or oversample weekend and urgent-billing examples; audit the label definition and replace the 'resolved quickly' proxy with an urgency ground truth (dispute value, refund risk, contractual SLA). Where ground truth is unavailable, have agents label a stratified sample.
Modeling: Add interaction features or let the model learn day-of-week × tier interactions; consider a per-slice model or a hierarchical approach if weekend volume stays low. Remove or de-weight features that encode staffing artifacts unless they are legitimately predictive of urgency.
Evaluation: Report precision/recall and calibration per slice (weekend/weekday, tier, ticket category), not just aggregate accuracy. Set an explicit acceptance threshold for the weekend-urgent slice (e.g., recall ≥ 0.90) before shipping.
Deployment safeguards: Shadow-mode the new model, route low-confidence weekend tickets to a human queue, and add a monitoring alert on weekend misroutes. Log routing decisions with model version for audit.
Success metrics: Weekend-urgent recall and median time-to-first-response for that slice, tracked weekly against a pre-deployment baseline; also monitor overall accuracy to catch regressions elsewhere.