The +1.2 pp That Wasn't Significant

Required sample. For a two-proportion test at alpha 0.05 (two-sided) and 80% power, n per arm is approximately 16 x p(1-p) / delta^2 = 16 x 0.61 x 0.39 / 0.000225 = 16 x 0.2379 / 0.000225, about 16,900 per arm, so roughly 33,800 sessions total. At 12,000 sessions per week that is about 2.8 weeks.

Where they are. Nine days at 12,000/week is about 15,400 sessions, roughly 7,700 per arm, which is 46% of the requirement.

What that means. Standard error on the difference at n = 7,700 per arm is sqrt(2 x 0.2379 / 7,700), about 0.79 pp. The 95% interval around +1.2 pp is therefore roughly [-0.35, +2.75] pp. The minimum detectable effect at this sample size is about 2.2 pp, so the experiment was never capable of resolving the 1.5 pp threshold they care about.

Interpretation. The result is consistent with no effect and with an effect well above the shipping threshold. p = 0.08 is not evidence of absence, and it is not evidence of a 1.5 pp gain. The honest read is that the test is unfinished.

Three process defects, all of which inflate the chance of a false positive:

1. Peeking with a fixed-horizon test. Three dashboard looks plus a plan to check daily against an alpha 0.05 fixed-sample threshold inflates the type I error rate substantially - commonly to the 15-20% range with several looks. Either declare a sequential design with an alpha-spending function or fix the horizon and do not look. Watching daily and stopping at the first crossing is the classic way to ship noise.
2. The guardrail failed and nobody is treating it as a failure. p95 latency rose 73%. If it was declared a guardrail, it was defined as a constraint, and a constraint breach is a stop condition, not a footnote. If the team is willing to trade 800 ms for 1.5 pp, that trade should have been written down before the test, not negotiated after seeing the point estimate.
3. The primary metric is a proxy. Task success inferred from action logs is not the same as user satisfaction, and it is sensitive to UI and logging changes. Pair it with a directly observed metric on a subsample, and treat any segmentation by user type or query topic as exploratory unless it was pre-registered - with this sample size, sliced cuts are noise generators.

The pattern to name out loud: the analysis is being shaped by the deadline rather than the other way around.

Do not ship to 100% on this evidence, and do not kill the reranker either. The data cannot distinguish the outcome from zero, and the guardrail is breached, so a full launch is a bet on a point estimate the design cannot support.

A defensible path under deadline pressure:

• Ship to a held-back slice, not the whole base. Keep a randomized 10-20% control permanently for two to four more weeks. Continue the measurement instead of abandoning it, and pre-register the stopping rule now so the next decision is not another negotiation.
• Fix the guardrail first if possible. If caching or a smaller reranker can recover most of the 800 ms at similar quality, the trade becomes cheap and the decision gets easier. If not, state the accepted cost explicitly and get it signed off.
• Pre-commit to the threshold. Success requires the lower bound of the interval to clear the agreed effect, not the point estimate.
• Define rollback triggers in advance: latency p95 above an agreed ceiling, or success dropping below baseline in the holdout for two consecutive days.

What not to do: present +1.2 pp at p = 0.08 as a win, extend the test while continuing to peek, or ship to everyone and call the follow-up monitoring a safeguard. Monitoring detects damage; it does not make an underpowered decision correct.