The Agent Chose the Prompt Template and Invented a Case Status

Root cause: overlapping descriptions plus a zero-cost failure mode.

All three actions claim the same job. Draft_Status_Reply literally advertises itself as the action to use "to answer questions about case status" - that is a superset of what the Flow action claims. To a reasoning layer choosing among candidates, the prompt template looks like the most general and most directly responsive option, and it always 'succeeds' because it has no external dependency to fail.

Structural fixes, not wording tweaks:

1. Remove the prompt template from the topic. A composition step belongs inside the flow path, not as a sibling the agent can select. Rename it to make its role explicit (e.g., Compose_Status_Text_From_Retrieved_Record) and call it from the Flow, or expose it only after retrieval succeeds.
2. Rewrite the Flow action description around the evidence it returns, not around the user question: "Returns the current status, last-updated date, and assigned office for one case number. This is the only source of case status."
3. Give the Apex callout a narrow, disjoint description (mainframe eligibility verification only) so it stops competing for status requests.
4. Add action-level preconditions: format validation on the case number; on validation failure, ask the customer rather than proceeding.
5. Add a grounding guardrail in the agent's instructions: never state a status, date, or processing stage that did not come from a tool response; if no tool returned data, say so and offer escalation.

Why this works better than stronger instructions alone: instructions compete with descriptions at selection time. Removing the plausible-but-unbacked option changes the choice set; instructions alone leave the bad option in the menu.

Ordering the agent should follow (expressed as agent instructions):

1. Collect a case number. If the customer supplies one, validate format. If validation fails, ask once more, then offer escalation - do not attempt retrieval.
2. Call Get_Case_Status with the validated number.
3. If the action returns not-found, say the case was not found and offer transfer. Do not speculate about why.
4. If the action returns a record, compose the reply from returned fields only. Cite the last-updated date.
5. If the customer disputes the returned status, stop and escalate - a dispute is not an FAQ.

What must NOT be left to the agent:

• The validation rule itself. Format checks and checksum logic belong in the Flow or a validation rule, not in prose the model may reformulate. The agent should receive a boolean, not a regex to interpret.
• The decision to answer without a tool result. Make this a hard constraint in instructions and test for it explicitly; it is the failure mode that produced the incident.
• Whether a status is 'final' or 'reviewable'. Stage semantics are business logic; expose them as enumerated values from the system, not adjectives the model chooses.

Error branch to define explicitly: if Get_Case_Status times out or errors, the required behavior is a plain statement that the system is unavailable plus an escalation offer. A silent fallback to the prompt template is the worst possible design - it is exactly the invented-answer path, just reached by exception instead of selection.

The gap: the pre-release test asked 'did the agent answer?' It never asked 'did the agent use a tool?' Fluency was mistaken for correctness.

Test design:

• Utterance set (n ≥ 30) stratified across: valid case number, malformed number, valid-format/nonexistent number, ambiguous ('where's my case at?'), multi-intent ('check my case and tell me about SNAP'), and adversarial ('just guess if you have to').
• Assertions per case:
  • Action selection: Get_Case_Status invoked exactly once; Draft_Status_Reply never invoked as a top-level action.
  • Grounding: every status token in the response appears in the tool payload. This is checkable mechanically - diff the response against the returned fields.
  • Failure behavior: malformed input produces a clarifying question, not a retrieval attempt and not an invented status.
  • Tone/accuracy: no date arithmetic the tool did not provide (a common invented-answer tell: 'within 5 business days').
• Pass criteria: 100% on grounding and on the never-invoke-prompt-template assertion; ≥ 95% on action selection across the stratified set; zero invented status tokens. These are not statistical targets - an invented benefit status is a compliance event, not a quality score.
• Regression fixture: add every failure found in production to the suite permanently, including this one.

What I would not accept as a pass: a green run on 5 happy-path utterances, or a subjective 'looks good' review by the caseworker who reported it. The defect was invisible to a reader who did not have the record in front of them - which means the test must compare against the record.