The Classification Eval That Can't Tell Good From Lucky
You are shipping a model that classifies customer messages into 12 policy-violation categories (some rare: category 11 appears in 0.3% of traffic). Your eval set is 300 examples, labeled by one contractor. The candidate model scores 94% overall accuracy vs 89% for the incumbent. On category 11, the candidate scores 100% (it got the 1 example right) and the incumbent 0%. The team wants to ship. Separately, you learn the eval set was sampled from the same week as the training data, and the contractor labeled 40 examples in under 10 minutes. The product owner asks: "94 vs 89 - isn't that clearly better?"
1. Explain why 94% vs 89% is not yet a shippable result, using the specific numbers.
Read the worked answer
Break the headline number apart and it stops meaning much.
The category-11 result is noise. At 0.3% prevalence, a 300-example set contains ~1 category-11 example. "100% vs 0%" is one coin flip. The confidence interval on 1/1 is enormous - you cannot distinguish a great classifier from a lucky one. Reporting it as a win is the classic rare-class trap.
The overall gap may also be within noise. 94% of 300 = 282 correct; 89% = 267. The difference is 15 examples. On a 300-item set, a 5-point gap is plausibly sampling variation depending on how the errors distribute across categories; you need per-category counts and CIs, not a blended percentage. A single blended accuracy also hides which categories got worse - the candidate could be better overall and worse on a high-severity category, which is the trade you actually care about.
The eval set is compromised in two ways. (1) Sampled from the same week as training data → possible leakage/overlap, inflating both models but not necessarily equally. (2) One contractor, 40 labels in under 10 minutes → ~15 seconds per label on a 12-way policy task; that's not enough time to read and apply a policy rubric, so label noise is likely and probably uneven across categories.
So: the comparison is underpowered on rare classes, possibly leaked, and labeled by a rushed single annotator. "Clearly better" is not supported.
2. Design the eval you would actually trust enough to ship on.
Read the worked answer
Build for per-category resolution, label quality, and leakage control - the three things the current eval lacks.
Sampling & size:
- Sample across a time window that does not overlap training data; dedupe against training by near-duplicate check.
- Stratify so every category has enough support to estimate. For category 11 (0.3%), oversample to at least ~50 examples so a per-category recall estimate has a usable CI. Report both the stratified metric and a prevalence-weighted metric so you don't fool yourself in the other direction.
- Size the set from the effect you need to detect; 300 is too small for 12 categories with rare ones.
Labeling:
- Multiple annotators on a subset; measure inter-annotator agreement (e.g., Cohen's/kappa). Low agreement on a category means the rubric is ambiguous - fix the rubric before trusting the model.
- Adjudicate disagreements; use the adjudicated label as gold.
- Audit the rushed contractor's labels; time-per-label is a red flag worth a re-review.
Metrics:
- Per-category precision/recall with CIs, not blended accuracy.
- A confusion matrix to see which categories get swapped (policy categories often confuse in pairs).
- Weight by severity: a false negative on a high-severity category costs more than on a low one; encode that in the ship criterion.
Ship criterion: candidate must not regress on any high-severity category beyond its CI, and must beat incumbent on the weighted metric with a CI that excludes zero. Only then is 94 vs 89 a decision.