The Support Triage Agent That Agrees With Everyone
You run a support triage agent on Claude. Tickets arrive via webhook; the agent reads the ticket, classifies severity (P1-P4), and for P1s drafts an escalation note to the on-call engineer. In shadow mode it agrees with human triage 91% of the time - but on the 47 tickets humans marked P1, the agent agreed only 38% of the time. When you inspect the drafts, the agent's escalation notes are fluent but often hedge: "this may warrant review." The system prompt says: "Be helpful and balanced. Classify severity. If unsure, prefer the less disruptive classification. Draft escalation notes when appropriate." Average ticket is 1,200 tokens; the agent also receives the last 20 messages of the customer's thread regardless of relevance. Latency p95 is 4.1s, within budget. Humans say the agent "reads like it doesn't want to bother anyone."
1. Diagnose the primary cause of the P1 recall failure and propose the highest-leverage prompt change.
Read the worked answer
The failure is a specification conflict, not a capability gap. The prompt contains three instructions that all push in the same direction on exactly the cases that matter most:
- "Be helpful and balanced" - no operational definition.
- "If unsure, prefer the less disruptive classification" - an explicit tie-break toward under-classification. P1 is the most disruptive class, so this rule systematically suppresses P1s.
- "Draft escalation notes when appropriate" - leaves "appropriate" undefined, and the "balanced" framing bleeds into tone, producing hedged notes.
The 91% overall vs 38% P1 split is the signature: the agent is fine on the easy majority and fails precisely where the tie-break rule fires.
Highest-leverage change: replace the tie-break with a severity-first rule and define P1 operationally. For example: "P1 means data loss, security exposure, full outage, or a paying customer blocked from core workflow. When a ticket plausibly meets any P1 criterion, classify P1 - escalate-and-downgrade is cheap; missing a P1 is not." Then give 3-4 concrete P1 and near-P1 examples. Separately, drop "balanced" and specify escalation-note tone (direct, states impact, states requested action, no hedging verbs).
Do not start by adding chain-of-thought or a bigger model - the evidence points at the rubric, and you'd be tuning around a spec bug.
2. You also suspect the 20-message thread window is hurting. How would you test that hypothesis without confounding it with the prompt fix?
Read the worked answer
Run the changes as two separate arms, not one combined edit:
- Arm A: prompt fix only, context unchanged (20 messages).
- Arm B: prompt fix + context change.
- Control: current production prompt.
Hold the eval set fixed: the same ~200 historical tickets, stratified so the 47 known P1s are all included plus a matched sample of P2-P4. Score on P1 recall, P1 precision, and a blind human rating of escalation-note quality. Because the P1 stratum is small, report confidence intervals - 47 tickets cannot resolve a 5-point difference; you'll need to accumulate more P1s or accept wide error bars.
For the context arm specifically, the mechanism matters. A 20-message window is not "more context is better" - it's often stale context (resolved sub-issues, earlier politeness) that dilutes the signal. A better design than a fixed window is a relevance filter: include the first message (problem statement), the most recent 3-5 messages, and any message a human later tagged as containing impact details. Test that against the flat 20-message baseline.
If Arm B beats Arm A on P1 recall beyond the CI, the context change earns its keep; if not, you've avoided shipping a more expensive, harder-to-debug pipeline for no measured gain.