The 'AI' Ticketing System That Never Learned
Priya manages a 12-person customer support team at an online garden-supply retailer. The company's ticketing system was sold as an 'AI assistant'. In practice, it assigns each incoming email to one of nine queues by scanning for keywords such as 'refund', 'broken' or 'delivery', using a list that Priya's predecessor wrote by hand. It never changes unless someone edits the list. Last month 18% of tickets were misrouted, mostly messages about damaged plants that mentioned 'delivery'. A vendor now offers a replacement that learns from 40,000 past tickets and agents' corrections. Priya has budget for one upgrade and must explain to her director whether the current system is AI at all, and whether the new one would behave differently.
1. Is the current ticketing system an example of AI? Justify your answer using the distinction between fixed rules and learned behaviour, and explain what evidence would settle the question.
Read the worked answer
Define the test by behaviour, not marketing. A system is rule-based if its behaviour is fixed by human-written conditions; it is learned if behaviour is derived from data and changes with it.
- Current system: nine queues and keyword lists written by hand. The mapping from email word to queue is deterministic and never changes unless a person edits it. That is automation, not machine learning. Many definitions of AI include hand-built expert systems, so the honest answer is 'narrow, non-learning automation' rather than 'not AI at all'.
- New system: trained on 40,000 labelled tickets, so its routing rules are inferred from examples and change when retrained.
Evidence to settle it: vendor model documentation; a test with paraphrased tickets that contain no keywords; whether performance improves after corrections; whether the same input can change over time. Priya should not accept the label on the invoice as evidence.
2. The new system learns from past tickets. What should Priya check before trusting it, and why might its reported accuracy on historical data overstate real performance?
Read the worked answer
Accuracy on historical tickets is the wrong test. The 18% misroute rate means some historical labels are wrong, so the new model may learn the predecessor's keyword habits. Tickets about damaged plants that mention 'delivery' are exactly the confusing cases; if they are spread unevenly across queues, overall accuracy can look high while the costly errors persist.
Checks:
- Hold out the most recent 3–6 months, not a random sample from the same period as training.
- Report per-queue precision and recall, and cost-weight errors: a misrouted refund request is slower than a misrouted brochure request.
- Test concept drift: new products, seasonal peaks, multilingual messages.
- Confirm human correction is captured and reviewed, and that agents can override.
- Ask for a monitoring plan and a rollback path.
Trust should be earned on unseen, recent data with the errors Priya actually cares about.
3. Priya's director says 'if it's not AI, we shouldn't pay for it'. How should she respond?
Read the worked answer
Respond that the label is not the purchase. The decision should be whether the replacement reduces the 18% misroute rate at an acceptable cost, not whether a vendor can call the old system AI.
- Reframe the business case in outcomes: agent minutes saved, refund delay reduced, customer satisfaction.
- Ask for a paid pilot on 4–6 weeks of live tickets with a control group, measured on the same per-queue metrics.
- Set exit criteria in advance and keep the hand-written system as fallback.
- If the pilot fails, the upgrade is not justified even if it is 'real AI'; if it succeeds, the old system's label is irrelevant.
This also protects Priya: she is buying a measured improvement, not a marketing category.