Stopping an A/B Test at Day 7: Peeking, Multiplicity, and Practical Significance
A retailer's experimentation team is testing a checkout redesign. The pre-launch plan specified 14 days, 50/50 allocation, and a single analysis at the end. Conversion is the primary metric; revenue per visitor, refund rate and support-ticket rate are tracked as secondary metrics, plus country-level breakdowns for six markets. At day 7, with 24,000 sessions per arm, control conversion is 1,008/24,000 (4.20%) and treatment 1,109/24,000 (4.62%); a two-sided test gives p = 0.025. Among the secondaries, revenue per visitor gives p = 0.04 in the favourable direction, refund rate gives p = 0.03 in the unfavourable direction, and support-ticket rate gives p = 0.61. Two of six country breakdowns show p < 0.05, both positive. The product owner wants to stop today and ship, arguing "p < 0.05, and the revenue metric agrees". Engineering estimates the rollout is worth doing only if the true lift exceeds 0.30 percentage points.
1. Is stopping at day 7 and shipping defensible? Assess the statistical validity of the day-7 p-value.
Read the worked answer
No, not as presented.
The design promised a single analysis at day 14. Looking at day 7 and acting if p < 0.05 changes the operating characteristics: each unadjusted look is another chance to cross 5%, so the family-wise error rate is no longer 5%. With repeated daily looks on accumulating data, the false-positive rate can rise well above 5% (typically into the 15–25% range depending on how correlated successive looks are).
So p = 0.025 cannot be read as "the 5% guarantee was met". The correct statement is that the observed difference is 2.2 standard errors from zero at an unplanned interim look.
If interim looks are genuinely needed, pre-specify a group-sequential design:
- Pocock boundaries hold a near-constant nominal level (roughly 0.029 per look for two equally spaced looks at overall 5%).
- O'Brien–Fleming boundaries spend little early (an interim threshold around 0.005 for a two-look design, with about 0.049 at the final look). The observed 0.025 would pass a Pocock interim boundary but fail an O'Brien–Fleming one - which shows how design-dependent the verdict is.
Check the mechanics too: exactly 24,000 vs 24,000 sessions is a clean 50/50 split, so there is no sample-ratio-mismatch warning. Had the arms drifted, the p-value would be untrustworthy before any of the above mattered.
Recommendation: do not stop now. Continue to day 14 and analyse once, or apply a pre-specified sequential boundary.
2. The team points to a favourable secondary metric as confirmation. Assess the multiplicity problem across the tests performed and recommend a testing strategy.
Read the worked answer
There are ten tests on the table: one primary, three secondary, six country segments. At α = 0.05 with no adjustment, the probability of at least one false positive among the nine non-primary tests is 1 − 0.95⁹ = 37%, with an expected count of 0.45. Seeing one or two "significant" secondary results is exactly what noise produces, so a favourable p = 0.04 is weak corroboration.
Strategy:
- Let the pre-specified primary metric alone drive ship/no-ship. Secondaries are supporting evidence and should be reported with effect sizes and confidence intervals, not used as tie-breakers.
- Gatekeeping: test secondaries only if the primary succeeds, then apply Holm or Bonferroni within the secondary family (α = 0.0167 for three tests). Revenue per visitor at p = 0.04 does not survive.
- Country breakdowns are exploratory subgroup analyses. Two of six significant is unremarkable (expected 0.3 false positives under the null); report per-market effects with intervals and flag the analysis as hypothesis-generating.
- Agreement between metrics is not independent confirmation: conversion and revenue per visitor are computed on the same sessions and are positively correlated, so they tend to move together even under noise.
The refund rate at p = 0.03 in the wrong direction deserves attention rather than dismissal. Multiplicity cuts both ways: a single nominal result is weak evidence, but a safety-type metric moving adversely is a risk signal to investigate with its confidence interval and, if possible, more data.
Recommendation: pre-register the primary, gate the secondaries, and treat segments as descriptive.
3. Compute the effect size and its confidence interval. What do they imply given the 0.30 percentage point threshold engineering quoted?
Read the worked answer
Pooled proportion: p̄ = (1,008 + 1,109)/48,000 = 2,117/48,000 = 0.04410.
Standard error of the difference: SE = √[p̄(1 − p̄)(1/n₁ + 1/n₂)] = √[0.04410 × 0.95590 × (2/24,000)] = √(0.04216 × 8.333 × 10⁻⁵) = √(3.513 × 10⁻⁶) = 0.00187.
Observed difference: 4.6208% − 4.2000% = 0.421 percentage points. z = 0.421/0.187 = 2.25, two-sided p ≈ 0.025, which reproduces the reported value.
95% CI: 0.421 ± 1.96 × 0.187 = 0.05 pp to 0.79 pp (relative lift 1.2% to 18.7%).
Interpretation:
- The data are compatible with an improvement as small as 0.05 pp (commercially nothing) and as large as 0.79 pp (clearly worthwhile). The point estimate of 0.42 pp overstates certainty.
- The lower bound of 0.05 pp sits below engineering's 0.30 pp threshold, so the test has not established that rollout is worth doing; it has also not ruled it out.
- Because this was an unplanned interim look, the nominal interval understates uncertainty further.
Decision framing: "promising but not established". Either continue to the planned day 14 (which also narrows the interval) or, if a decision is needed now, state that the evidence is compatible with both a trivial and a substantial effect. If the organisation requires a minimum detectable effect of 0.30 pp, size the next experiment for 80% power at that effect rather than relying on this one; with baseline 4.2%, detecting a 0.30 pp lift needs roughly 4 × 24,000 sessions per arm for comparable power.