One Mega-Topic, 38 Actions: A State Licensing Board's 31% Misroute Rate

Diagnosis: this is an information-architecture defect, not a reasoning-capability defect.

The evidence points away from the model:

• Errors are systematic and directional (wrong action, wrong data request), not random or shallow. A stronger model reasons over the same ambiguous menu and will not reliably disambiguate 38 actions bundled under one label.
• The complaint/renewal confusion is a scoping error. If complaint intake lives inside a topic whose instructions are written for renewals, no amount of extra reasoning fixes the conflict - the instructions themselves contradict the desired behavior.
• The 200-word monolithic instruction block cannot express per-action preconditions, so the agent has no way to know that a license number is required in one path and irrelevant in another.

What I change first, in order:

1. Split into three topics - Renewal, New Application, Complaint Intake - each with a description written in the user's vocabulary and a negative boundary ("not for complaints about a business you are not licensed by").
2. Move to action-level instructions: each action declares when it applies, what inputs it requires, and what it must never do (e.g., renewal status action: "requires a validated license number; never call before validation succeeds").
3. Delete or demote actions that overlap. 38 actions in one topic almost always includes near-duplicates; merge them before adding anything.
4. Re-run a stratified test set per topic before touching escalation thresholds.

On the proposed changes: a second reasoning pass adds latency and cost against an unresolved ambiguity, and raising the escalation threshold to 0.4 pushes ambiguous chats into autonomous resolution - precisely where the complaint/renewal confusion is most damaging. I would hold both until misroute is measurably down, then tune thresholds against a labeled set.

Criteria I apply, in priority order:

Criterion	Leans agentic	Leans deterministic (flow/guided path)
Entry-point ambiguity	User's need must be inferred from free text	User's need is a known menu of 3–5 options
Consequence of error	Low: read-only info, FAQ, explain status	High: creates a case, changes a record, starts an SLA clock
Required field set	Variable, discovered conversationally	Fixed and legally mandated
Regulatory defensibility	Explanation can be free-form	Same questions must be asked, in the same order, to everyone

Applied to this org:

• Complaint intake → deterministic. Fixed field set, statutory clock starts on submission, and duplicates are already the visible failure. Free-text intake plus agent discretion is the wrong tool. Let the agent recognize the intent and hand off to a guided flow, retaining the conversational entry point.
• New application → hybrid. Agent handles orientation, document checklist, and 'where am I' questions; the submission itself is a deterministic flow with server-side validation.
• Renewal status → agentic, read-only. High volume, low consequence, high variation in phrasing. This is where autonomy earns its keep.

The principle: keep the conversational surface agentic and the state-changing surface deterministic. The agent's job is to understand and route, not to be the system of record.

Why raw containment is the wrong target here: containment counts a chat as a win whether the customer got the right answer, an invented one, or gave up. Optimizing it directly incentives the agent to guess rather than escalate - the exact behavior the complaint flow is already exhibiting.

Metric set I would propose:

• Verified containment - resolved without escalation and no reopen, no repeat contact within 7 days, no negative CSAT. This is the headline number.
• Misroute rate - % of conversations executing an action outside the labeled intent. Currently 31%; this is the diagnostic that must fall first.
• False-confidence rate - cases where the agent asserted a status or outcome with no tool output behind it. Sampled weekly from transcripts; target is zero, not 'low'.
• Escalation appropriateness - of escalations, the share that genuinely required a human. Punishes both under- and over-escalation.
• Duplicate-case rate - directly attributable to the current defect.

Escalation policy, stated as rules rather than a single confidence number:

1. Escalate whenever a required input fails validation twice.
2. Escalate whenever the customer disputes a status the agent retrieved (status disputes are a signal, not an FAQ).
3. Escalate any complaint-intent utterance that reaches the renewal topic - this should be structurally impossible, but treat it as a tripwire while the split beds in.
4. Never let a confidence score alone authorize a state-changing action; require validation success.

A single 0.4 threshold conflates 'the model is unsure' with 'the situation is consequential.' Those need separate controls.