Predictive Intelligence on a Thin, Imbalanced Incident Sample
Northwind Health has 40,000 open and closed incidents in one ServiceNow instance. Leaders want Predictive Intelligence to auto-set category and assignment group before agents touch new tickets. The training extract has only 300 labeled incidents: 210 Network, 45 Hardware, 30 Access, and 15 Software. The service desk creates about 2,800 incidents per month, and the backlog is growing 12% quarter over quarter. Two service desks use different definitions of “Network,” and neither has a documented taxonomy. The platform team has one part-time data analyst, no dedicated ML engineer, and a mandate to show value in 60 days. Agents currently re-categorize about 35% of tickets because the first selection is wrong. The CIO asks whether the model can be trained next week and switched on for all four categories at once.
1. Should the team train and enable all four categories next week? Assess the data, model, and change-management issues, and recommend a first release.
Read the worked answer
No - do not train and enable all four categories next week.
- Data volume and balance: 300 labeled records is a thin base, and the per-class counts matter more than the total. Software has 15 examples, which is not enough to learn a stable pattern; Network is 70% of the sample, so a model can look accurate by predicting Network and still fail the minority classes.
- Label quality: The two desks define “Network” differently. Training on inconsistent labels teaches the model the disagreement, not the work. Fix the taxonomy and relabel a stratified sample before training.
- Evaluation: Overall accuracy is the wrong metric. Use per-class precision/recall and a confusion matrix on a held-out set, with a minimum support threshold per class.
- First release: Start with the one or two categories that have enough examples and clear business value, run in shadow mode or suggestion-only mode, and keep a human decision. Add agent feedback and review a sample weekly.
- 60-day value: Define value as reduced re-categorization on the pilot categories and faster routing, not full automation. That is credible with one part-time analyst and no ML engineer.
2. What controls would keep the model from degrading after go-live, and what would you do with the agents’ re-categorizations?
Read the worked answer
Treat agent corrections as a feedback signal, not automatic ground truth. Agents sometimes pick the first available category under pressure.
Controls:
- Capture the original prediction, the agent’s final choice, and a reason code; review a sample before adding to training data.
- Maintain a frozen holdout set and re-score each retrain; watch per-class precision/recall, not just volume.
- Set a retrain cadence, for example quarterly or when drift exceeds a threshold, and assign a named owner for the model, taxonomy, and label quality.
- Monitor input drift: new products, acquisitions, or channel changes can shift language.
- Exclude high-risk incidents such as security, P1, and regulatory cases from auto-assignment; route them to humans.
- Keep a fallback: if confidence is low, leave the category blank and ask the agent.
- Version the model and document changes for audit.
3. How would you frame the 60-day value story to the CIO without overpromising?
Read the worked answer
Frame it as a controlled pilot with a measured baseline, not a platform-wide AI launch.
- Baseline the current 35% re-categorization rate, average handling time, and misroute rate for the pilot categories.
- Promise a target for the pilot: for example, reduce manual re-categorization on the selected categories by a defined percentage, with agent acceptance measured weekly.
- Show leading indicators in 30 days: suggestion acceptance, label quality, taxonomy agreement. Show outcome indicators in 60 days: rework and routing time.
- Be explicit about what is not included: no auto-assignment for all categories, no P1 automation, no headcount claim.
- Report the cost of data preparation and the risk that minority categories will take longer. That gives the CIO a defensible value story without overpromising.