Retail Bank Digital Assistant Misroutes Card PIN Requests
A retail bank has a digital assistant named RetailBank with three skills: Accounts, Cards, and Loans. The assistant handles about 18,000 sessions per month across web and mobile. Each skill contains 10–15 intents; skill-level confidence thresholds are set to 0.65. Users report that questions such as 'I need a new PIN for the card ending 4471' often open the Accounts flow and ask for an account number. Logs show 22% of card-PIN conversations end unresolved. The Cards skill has a 'Reset PIN' intent with 14 training utterances, while Accounts has a broad 'Account help' intent with 90 utterances. The team wants to add a new Loans intent for 'hardship application' next sprint and is debating whether to raise all thresholds to 0.8. The digital assistant was last published three weeks ago; the draft has since been edited by two builders.
1. Explain why 'I need a new PIN for the card ending 4471' routes to Accounts even though Cards has a Reset PIN intent. What evidence would you collect before changing anything?
Read the worked answer
Routing is not keyword matching. When a message arrives, ODA scores it against intents in every skill and any digital-assistant-level intents. The highest score above the applicable threshold wins. Accounts' broad 'Account help' intent has 90 utterances and vocabulary such as 'card', 'new', and 'ending', so it can outscore Cards' 'Reset PIN' intent with only 14 utterances. The phrase 'card ending 4471' is an entity-like detail, but routing can occur before entity resolution; failure to resolve an entity does not prevent a skill from being selected. At a 0.65 threshold, an unimpressive Accounts score can still pass.
Before editing, collect: the exact utterance run in Utterance Tester with all skills visible; per-intent scores and the winning skill; the entity resolution trace; whether the tester is using draft or published content; 50 recent misrouted conversations with expected skill/intent labels; and overlapping training utterances between Accounts and Cards. Also check the publish date. If production runs the version from three weeks ago, draft edits are not live and any diagnosis based on draft may mislead.
2. Propose a remediation plan covering training data, digital-assistant-level routing, thresholds, and skill design. Explain why raising every threshold to 0.8 is a poor default.
Read the worked answer
Remediation should be targeted:
- Add discriminating utterances to Cards' Reset PIN intent: PIN arrival, lost or stolen card, activate new PIN, card ending digits. Remove or narrow ambiguous 'card help' utterances in Accounts so it stops absorbing card-PIN language.
- Add digital-assistant-level routing intents for high-risk cross-skill phrases if the assistant version supports them, and route those explicitly to Cards. Do not rely on one broad Accounts intent to cover everything.
- Tune thresholds selectively. Raising all skills to 0.8 will increase unresolved conversations and reduce containment; a high threshold only helps precision when the training data is already separable. Set per-skill thresholds from a labeled regression set, not as a global policy.
- Add a card-last-four entity in Cards and prompt for it there instead of letting Accounts ask for an account number. Consider a disambiguation prompt when the top two skills score within 0.05 of each other.
- Keep the Loans hardship intent out of the same release unless it is included in regression testing; new training data can shift routing scores.
The goal is to make Cards the obvious winner for card-PIN language while preserving recall for legitimate Accounts requests.
3. How would you validate the routing change and manage the draft/publish gap before and after release?
Read the worked answer
Build a regression set of at least 120 utterances drawn from logs, including 20 adversarial card-PIN utterances, 20 account utterances, and 10 near-miss loan utterances. Run them in Utterance Tester against the draft with all skills enabled and record the winning skill and score margin. Target: Cards Reset PIN is top for at least 90% of card-PIN utterances, Accounts misroute is under 5%, and unresolved is under 10%.
Test in a separate test digital assistant with the same channel configuration, not only in the skill tester, because assistant-level routing and channel behavior differ. Clone or version the assistant, publish to test, and run end-to-end smoke tests on web and mobile. After production publish, monitor unresolved rate, skill handoff, containment, and misroute complaints for two weeks against the 22% baseline. Keep the prior version available for rollback. Do not let builders continue editing the draft during final validation, or the tested artifact will not match what is published.