Canary Gate: A 1.3-Point Regression on 1,200 Ratings

Take them one at a time.

• ROUGE-L, +0.4 points. Decision-grade in neither direction. A 0.4-point move on 1,200 samples is well inside run-to-run variance for this metric, and ROUGE-L is largely insensitive to whether a number like a disputed amount survives into the summary. It cannot clear the gate; treat it as flat.
• p95 latency 2.1 s vs 1.9 s. Real but modest, and plausibly a length artifact. Worth a gate only if there is an SLO behind it.
• Cost per request +18%, driven by longer output. Decision-grade and directly attributable: more output tokens, more decode, more money. This is a fact, not a statistic.
• Thumbs-down 4.2% vs 2.9% on n=1,200. Standard error of the difference: sqrt(0.042x0.958/1200 + 0.029x0.971/1200) ≈ 0.75 points. The 1.3-point gap is ~1.7 standard errors, roughly 95% CI from about -0.2 to +2.8 points. Not significant at the 5% level. On its own, this does not justify a rollback.
• Billing-dispute escalations. Qualitative but the strongest signal in the table, because it is a mechanism: a specific failure (dropped disputed amount) on a specific slice, which is exactly the class of error aggregate ROUGE-L is blind to.

Hold the canary at 10% - do not promote, and do not roll back yet. Six hours is enough to replace an uninformative decision with an informative one, and a full rollback has its own cost and churn.

The reasoning: the promotion gate says "no significant regression". Nothing significant has been demonstrated - but the thumbs-down interval spans zero in one direction and reaches +2.8 points in the other, so the data is simply too thin to declare safety, and the escalations describe a concrete failure mode the metrics cannot see. Promoting on "ROUGE went up 0.4" would be promoting on noise while ignoring a named defect.

Six-hour plan:

1. Slice the data now by ticket category. If billing-dispute thumbs-down is materially above baseline on a few hundred samples, that is a defensible, mechanism-backed regression and the gate fails on evidence rather than vibes.
2. Length-stratify. Check whether the escalation cluster is v3 producing longer summaries that drop specifics. If so, cap max output tokens and re-run - a cheap fix that also recovers much of the 18% cost.
3. Pull 50 escalations and have a human label whether the disputed amount is present. That is a direct measurement of the thing that matters, available in hours.
4. Freeze promotion until either the slice regression is confirmed (then roll back and fix) or it is shown to be absent (then promote with a documented caveat).

Do not let the timer force a decision that the data does not yet support; extend the canary if the slice work needs another day.

The gate did not work - it produced a metric that disagreed with reality and a timer that nearly forced a decision on noise. Required changes:

• Slice-level gates by default. Any aggregate metric must be accompanied by per-category results with per-slice sample sizes. A single dominant slice should be able to fail the gate on its own.
• Length-controlled quality metrics. v3 emits longer summaries, which mechanically inflates recall-oriented metrics like ROUGE-L. Gate on a metric that controls for length, or report length directly alongside it.
• A metric that can see the defect. For billing content, track whether key entities (amounts, dates, account IDs) survive into the summary - an entity-preservation rate, not n-gram overlap. That is the failure the support queue caught and the dashboard missed.
• Power the thumbs-down gate. n=1,200 detects roughly a 2-point absolute change. If the product cares about 1-point shifts, either sample more ratings or pre-declare that thumbs-down is a leading indicator only, with a human review trigger.
• Cost and latency as first-class gates, not footnotes - the 18% cost increase was the only unambiguous, decision-grade signal in the entire table, and the policy did not name it.
• Pre-registered decision rules, including what to do when the canary expires with inconclusive data (extend vs decide). The six-hour cliff should never be the thing that decides.