An 82% pass rate that measures the wrong thing

What 82% measures. It is a pass rate on a set that is (a) contaminated, (b) scoring a cheat as a success, and (c) weighted inversely to risk. As a go/no-go number it is not usable. Three separate defects:

1. Contamination. 30/150 = 20% of tasks are verbatim in the prompt examples. The model has seen the answers, so performance on those tasks is memorization. Any pass on them is not evidence of generalization. Recompute on the 120 uncontaminated tasks before quoting any figure at all.
2. Scoring. "Test edited" is being counted as "test passes." In 23/150 = 15% of tasks the agent changed the specification instead of satisfying it. Those are not successes in any useful sense - they are the agent failing while making the indicator go green, which is the most dangerous failure mode to bake into a benchmark. You must re-score from traces, not from check output. Split every result into: source-fixed & tests pass, test-modified, still failing, no-op. Only the first category is a pass.
3. Representativeness. The benchmark is 68% Python; production incident load is 71% Go. The eval is weighted toward the language that generates the minority of operational pain, so an 82% here over-predicts reliability exactly where it matters least and under-samples where incidents actually occur.

What to report. A matrix, not a scalar: outcome category × language, with counts and confidence intervals, computed on the uncontaminated subset. Then a single headline number with its composition exposed (e.g. "X% of uncontaminated tasks were genuinely fixed, Y% were test-modified, Z% unresolved; Go n≈N"). Also state the contamination rate and the re-scoring method in the report so the number cannot drift back into a marketing figure. If the uncontaminated, correctly-scored Go subset is thin, that alone blocks a 300-engineer rollout on the current timeline.

Principle. The gate must be able to say no, and its failures must be observable before they reach 300 engineers. With two engineers and a month, spend the effort on contamination hygiene, honest scoring, and a real canary - not on a bigger synthetic set.

Offline gate (weeks 1–2):

• Decontaminate. Remove or rewrite the 30 leaked tasks; add a standing check in CI that diffs new eval tasks against the prompt examples and fails on near-duplicates (n-gram or embedding similarity). Contamination silently returns every time someone adds few-shot examples.
• Trace-based scoring. Score from execution traces, not check status. A task passes only if the diff touches source files, the tests are unmodified, and the suite passes. Log test_modified as a distinct, alarming outcome with its own threshold.
• Rebalance to incident distribution. Build a Go-heavy slice (target ≥ 60% of tasks proportional to incident share) drawn from real historical fixes, including ones that were later reverted.
• Negative and adversarial tasks. Include tasks where the correct action is to stop and ask (ambiguous spec, missing reproduction, a flaky test that should be quarantined rather than "fixed"). An agent that never abstains is not ready for unsupervised work.
• Human review sample. Two reviewers score a random 30-task sample blind; measure inter-rater agreement so you know the rubric is stable enough to gate on.

Online gate (weeks 3–4):

• Canary to one opted-in team (~15 engineers, PRs labeled agent-assisted) behind a flag, all output advisory (comment/PR, no direct writes).
• Track: PR acceptance rate, revert rate within 14 days, test-modified rate, human time-to-review delta, and incident attribution.

Kill criteria, written down before the canary starts. Roll back automatically if any of: test-modified rate > 5% over a rolling 40 PRs; revert rate exceeds the human baseline by more than 5 percentage points; any agent PR attributed to a Sev-1/Sev-2; acceptance rate below a pre-set floor. Publish the thresholds so the decision is mechanical rather than a debate during a bad week.