71% agreement on a six-way classification
A revenue-ops team classifies inbound sales emails into six pipeline stages plus a three-level urgency. The current prompt is a 400-word paragraph in a user message, followed by three few-shot examples that are all labelled Stage 3 / Medium. Temperature is 1.0 and the only quality instruction is 'classify as accurately as possible'. On a 500-email human-labelled set, the model agrees with the human label 71% of the time; two humans agree with each other 89% of the time. Two failure modes dominate: about 4% of outputs contain a seventh, invented label such as 'maybe Closed Won?', and Stage 2 versus Stage 3 confusions account for most of the rest. The team wants to swap in a larger model. You have one week and no budget increase.
1. What is wrong with the current prompt? Prioritise the changes you would make before considering a model swap.
Read the worked answer
The prompt is under-specified in four separate ways, and each maps to an observed failure.
1. The label set is not closed. 'Six pipeline stages' described in prose is an invitation to paraphrase. Enumerate the exact six strings, say 'the label must be exactly one of these strings', and provide the output shape. That alone should kill the 4% invented-label rate. Better still, constrain the output so an invalid label is structurally impossible.
2. The examples carry no boundary information. Three examples, all Stage 3 / Medium, teach the model a prior, not a task. They actively bias toward the majority class and give zero signal on the only boundary that matters (Stage 2 vs Stage 3). Replace them: 2–3 examples per confusable pair, deliberately including near-misses, and keep the class distribution roughly balanced so the examples aren't a second prior.
3. There is no decision rule for the hard boundary. Write an explicit, ordered rubric for Stage 2 vs Stage 3 - what evidence decides it, what to do when both are present. Vague instructions produce variance, and variance on a boundary is exactly where disagreement shows up.
4. Temperature 1.0 for a labelling task. Sampling temperature is adding noise to a deterministic decision. Lower it; note that low temperature reduces but does not eliminate non-determinism, so don't build tests that assume byte-identical output.
Also: move the rubric into system, delimit the email body with tags, and ask for the label followed by a one-line justification (or a confidence value) so disagreements are diagnosable rather than opaque.
2. How do you know a change is a real improvement rather than noise, given the 89% human-human ceiling?
Read the worked answer
Set the ceiling and the sample size first. Humans agree 89%, so ~11 points of your 29-point gap is annotation noise or genuine ambiguity - you cannot beat the ceiling and you shouldn't try. At n=500, a 71% → 74% move on the same items is well inside sampling noise if you treat it as two independent proportions; you need a paired comparison.
Design:
- Freeze a held-out test slice and never tune on it. Keep a separate dev slice for prompt iteration; otherwise you are fitting the prompt to the test set and the number becomes meaningless.
- Run old and new prompts on the same items, score per item, and use a paired test (McNemar on discordant pairs, or a bootstrap CI on the difference). 'It went up 3 points' with overlapping CIs is not a result.
- Stratify by class and report a confusion matrix, not just accuracy. Your dataset is presumably imbalanced, and the failure you care about (Stage 2 vs Stage 3) is invisible in an aggregate number.
- Track invalid labels as a separate hard metric - target zero. A validity failure is not an accuracy point, it's a broken contract downstream.
Define a ship criterion in advance, e.g. macro-F1 improves with a CI excluding zero, invalid-label rate is 0, and no class regresses more than 3 points. Then read the remaining Stage 2/3 disagreements by hand - if the humans can't agree on them either, the rubric is the problem, not the model, and no larger model will fix it.
3. The team wants to add chain-of-thought reasoning to every request for an accuracy boost. Evaluate that trade-off.
Read the worked answer
It probably helps the hard boundary and definitely costs money. Reasoning tokens are output tokens: they are billed, they add latency, and at volume they will dominate generation cost. Whether it clears the bar is an empirical question you can answer in an afternoon.
Test it as an experiment, not a default. Run three variants on the dev slice, same items: (a) label only, (b) label then one-line justification, (c) reasoning then label. Compare macro-F1 and output tokens per request. If (c) buys two points of macro-F1 at triple the output tokens, price that against the cost of the human review you're avoiding.
The cheaper architecture is usually a routed second pass. Label first in one short call. Attach a confidence value or a 'boundary flag' - e.g. the model must say when Stage 2 and Stage 3 evidence are both present. Send only that minority of items to a second, more expensive call with reasoning enabled, or to a human. You get most of the accuracy at a fraction of the tokens.
Two caveats. Justification written after the label tends to rationalise the label rather than justify it - if you want the reasoning to influence the answer, it must come first. And a confident-sounding justification is not evidence of correctness; if you use these rationales for audit, spot-check them against the email text, because a wrong label with a fluent reason is worse than a wrong label alone.