Triage Model That Learns to Say No
Northwind Managed Services handles 14,000 tickets a month for 38 clients with 620 agents. Eight months ago it deployed a vendor ML model that classifies each ticket's priority (P1–P4) and routes it to a resolver group. Overall accuracy is 93% and the vendor reports the model as healthy. But P1 recall is 0.71 against a human baseline of 0.94, and agent override rates have climbed to 38%, almost all on tickets from one logistics client whose volume triples every November. Mean time to restore for P1 incidents has risen from 3h10 to 3h32 since go-live. The data scientist argues that 93% accuracy beats the human baseline on the same measure. The operations director wants the model switched off for the logistics client and wants to claim an SLA credit from the AI vendor. The service integration manager is worried that nobody can explain, after the fact, why any individual ticket was classified the way it was.
1. The data scientist and the operations director disagree about whether the model is performing. Explain how you would evaluate whether the model is fit for purpose, and state what you would do in the next ten working days.
Read the worked answer
Overall accuracy is the wrong lens here. The class distribution is skewed and the cost of error is asymmetric: a P1 read as P3 costs hours of outage and possibly contractual penalties, while a P3 read as P1 costs a wake-up call. So evaluate per class and per cohort - build a confusion matrix segmented by client, service, season and ticket source. P1 recall of 0.71 means roughly three in ten major incidents are not being raised at major-incident speed; that justifies intervention regardless of the 93% headline.
Second, test for drift and for baseline contamination. The logistics client's November peak sits outside the training distribution, so recall degradation there is expected behaviour of a stationary model, not random noise. Check whether the human 0.94 was measured under the same seasonal conditions before treating it as a fair comparator.
Third, treat the 38% override rate as a signal rather than a nuisance: it is a labelled stream of disagreements that should be sampled weekly and fed into the improvement backlog.
Actions within ten working days: keep the model on, but add a deterministic rule that tickets containing the client's major-incident keywords are auto-escalated and human-reviewed before classification - a compensating control, not a rollback. Freeze the vendor's quarterly retrain until it supplies a client-segmented evaluation report. Commission a sample audit of 100 overrides. Record the decision, the evidence and the residual risk with the service integration manager so the audit trail exists. Switching off wholesale would forfeit routing benefit that is genuinely working for the other 37 clients.
2. The operations director wants to claim an SLA credit from the AI vendor. Who is accountable for the degraded MTTR, and how should the contractual and governance position be framed?
Read the worked answer
Accountability stays with Northwind. An AI system is a resource in a value stream, not an accountable agent; the logistics client contracted with Northwind, not with the AI vendor. "The model did it" is not a defence in the SLA conversation, and treating it as one damages the customer relationship more than the 22 minutes of MTTR does.
The vendor relationship is a separate supplier-management question with three parts. What performance commitment was actually written - accuracy on which dataset, measured how often, with what remedies? Was the training data representative of seasonal peaks, and did Northwind disclose the November pattern when onboarding? Was there a notification duty before the last retrain?
Practical framing: quantify the loss before arguing about 93% versus 94%. Incremental MTTR × P1 volume × contractual penalty exposure gives a number both parties can discuss. Then open a supplier performance review with an evidence pack - segmented confusion matrices, drift analysis, override samples - and ask for a remediation plan and a re-baselined acceptance test, using credits as leverage rather than the objective.
In parallel, add AI-specific clauses to the next contract: cohort-level performance thresholds, evaluation data supplied by Northwind, retraining notification, right to audit, and portability of decision logs on exit. Log the whole matter in the risk register with the operations director as risk owner and a review date, because until the compensating control is proven the residual risk is live.