Support ticket routing with a 22% fallback rate

Likely causes, in order of probability:

1. Class imbalance. Two intents hold 78% of utterances. Models learn the majority classes and output low confidence on the rest; 'Data export request' with 140 examples is effectively under-trained.
2. Language coverage skew. German fallbacks being over-represented suggests German utterances are few or phrased differently from training data. CLU can handle multilingual input, but it learns from what it is shown.
3. Threshold set globally. A single confidence threshold tuned for Billing will be wrong for rare intents.
4. Structural mismatch: tickets with two issues get one intent, so one issue is always dropped - which will look like a fallback or a misroute.

Measure first: a confusion matrix per language and per intent, plus the distribution of confidence scores for correct vs incorrect predictions. If German recall is fine but German confidence sits just under the threshold, the fix is a threshold change, not more data. Changing the model before this measurement is guesswork.

A probabilistic classifier cannot guarantee a floor on recall, so do not rely on CLU alone.

• Any single model has some false-negative rate. 'Never misrouted' is a recall guarantee; a scoring model provides recall estimates, not guarantees. Tuning a threshold trades GDPR recall against false positives on unrelated tickets.
• Design: a deterministic high-recall pre-filter ahead of CLU. In an orchestration workflow, run a rule/keyword layer (e.g. terms for deletion, erasure, 'Recht auf Löschung', 削除) on the raw text. Anything matching is force-routed to the legal-flagged queue regardless of CLU's prediction. Rules are auditable, which is exactly what Legal wants.
• Accept the cost: the pre-filter will over-capture - a ticket mentioning deletion of a saved report gets escalated. That false-positive rate is the price of the guarantee, and Legal should sign off on it explicitly.
• Add a safety net: a low CLU threshold for the GDPR intent (route to human review rather than auto-answer) and a human confirmation step before any automated action.
• Audit trail: log the matched rule and model confidence for every escalated ticket.

Split the problem: classification for triage, CLU for the conversational intent.

• CLU is built for single-turn intent assignment. Forcing multi-issue tickets through it guarantees one issue is dropped - that is a design limitation, not a training-data problem.
• Introduce a custom text classification (multi-label) model over the ticket text to tag all applicable categories (Billing, Password reset, Data export, GDPR). This answers 'what is this ticket about' and is evaluated with per-label precision/recall rather than a single accuracy figure.
• Keep CLU for the chat/voice channel, where the real job is resolving one intent per turn and following up.
• Handle imbalance directly: upweight or oversample the rare labels, and evaluate per-label. Aggregate micro-accuracy will stay high while GDPR recall quietly sits near zero.
• Backfill: since only 140 and 96 examples exist, mine production tickets with reviewer confirmation to grow these labels - the 22% fallback queue is yourself-labelled gold data.

Implication: the existing CLU project should not be deleted; it becomes one component in an orchestration workflow rather than the whole solution.