The Triage Prompt That Got Worse Every Time It Got Fixed
A SaaS company's ticket-triage prompt classifies 12 support categories and returns JSON. Over three weeks in production: 'Other' is returned 31% of the time although the labeled rate is 9%; roughly one in six responses arrives wrapped in Markdown code fences, breaking the parser; and urgency labels skew high on weekends. Every time an engineer sees a bad ticket, they add a corrective few-shot example and redeploy. Accuracy on the original 300-ticket development set has fallen from 84% to 79% across eleven such patches, and nobody can say which examples still help. The prompt is now 4,100 tokens of instructions and 18 examples. There is no held-out set, no inter-annotator agreement number, and the product manager wants a fix shipped this sprint.
1. Explain why measured accuracy fell while the visible error rate seemed to improve. Name the failure pattern and propose the structural fix.
Read the worked answer
Pattern: anecdotal patching with no held-out evaluation - an overfitting loop.
Each patch was fitted to one observed ticket, so the error the engineer remembered disappeared. Three things degraded invisibly:
- Example dilution. 4,100 tokens of instructions and 18 examples compete for attention; examples for recent edge cases push the common-case instructions further from the point of generation.
- Label drift. Corrective examples encode one engineer's judgment, which may not match the annotation guide, so the model learns a private taxonomy.
- Selection bias. Only failures get sampled, so the example set over-represents weird inputs and under-represents the traffic that actually arrives.
Dev-set accuracy fell because the prompt now describes many edge cases and few typical ones; the 84%→79% drop is the honest signal, and it is invisible day to day.
Structural fix:
- Stop patching. New failures go into a candidate pool, not the prompt.
- Build a stratified 400-ticket held-out set, never used for authoring, with 100 double-annotated to establish the agreement ceiling.
- Require every prompt change to clear a pre-declared bar against that set (e.g. +2 points overall, no category down more than 3).
- Cap the prompt at roughly 8 examples chosen for coverage of decision boundaries, not recency.
2. Redesign the prompt and the surrounding integration: where format constraints belong, how few-shot examples should be chosen, and how to guarantee parseable JSON.
Read the worked answer
Format is an API problem first, a prompt problem second.
- Enforce structurally. Use a tool/structured-output schema, or prefill the assistant turn with
{so a fenced code block cannot open. Add a post-parse validator with one repair retry. Do not spend prompt tokens persuading the model to avoid Markdown fences. - Instruction placement. Put the output contract and category definitions at the end of the system prompt, immediately before the conversation; the most recent instruction dominates. Keep role framing at the top.
- Category design. A 31% 'Other' rate against a 9% label rate means the 11 real categories are not mutually discriminable from their current definitions. Write one-line definitions plus a contrast rule for each confusable pair (e.g. dispute = contesting a charge; refund = asking for money back).
- Example selection. Choose by coverage of decision boundaries, not recency. Six to eight examples that each sit near a boundary beat eighteen typical ones. Hold the example set fixed between evaluations so numbers are comparable.
- Keep the variable part small. Ticket text goes in the user turn; the taxonomy and contract stay in the cached prefix so repeated traffic does not re-pay for them.
3. The product manager wants a decision now: ship this sprint or not? Define the minimum viable evaluation and the release gates.
Read the worked answer
Ship - but ship the eval scaffold plus one deliberate rebuild, not another patch.
Minimum viable evaluation, roughly two days:
- 400 held-out tickets, stratified by category, drawn from the last 90 days, none used for authoring.
- 100 of them double-annotated; report Cohen's kappa. If kappa is below 0.7, fix the annotation guide before trusting any number - you cannot measure a model against labels humans disagree on.
- Metrics: per-class F1, macro-F1, 'Other' rate, JSON validity rate, and a confusion matrix for the two worst pairs. Overall accuracy hides exactly the failures that matter.
Release gates: JSON validity 100% (hard gate); no category F1 down more than 3 points versus current production; macro-F1 up at least 3; 'Other' rate within ±5 points of the labeled base rate.
The decision to communicate: the rebuild (boundary definitions, contrast rules, structural output enforcement, example cap) fits in one sprint. Everything anecdotal gets logged, not shipped. If the rebuild fails the gates, production stays on the current prompt and you say so with evidence - that is far cheaper than eleven more patches.