Stopping a Live A/B Test at Look Nine
A news site's growth team is A/B testing a redesigned article header against the current one. The primary metric is article click-through rate, with a pre-experiment baseline of 4.00%. Traffic is split 50/50 and 3,000 users per arm arrive each day. The plan filed with the analytics lead specifies a fixed 14-day horizon, i.e. 42,000 users per arm. Six secondary metrics are tracked as well: scroll depth, video plays, newsletter sign-ups, comments, shares and ad viewability. On the morning of day 9, with 27,000 users per arm, the dashboard shows 1,193 clicks on the new header against 1,080 on the old, and a two-sided p-value of 0.015 on the primary metric. The product manager wants to ship that afternoon. Two details surface afterwards: users with fewer than two page views in their session were dropped from the analysis starting on day 4, and newsletter sign-ups show p = 0.03 in the same read-out.
1. Recompute the day-9 two-proportion test from the raw counts and state the effect size with an interval. Does a p of 0.015 license shipping under the filed design?
Read the worked answer
Pool the arms: p̂ = (1193 + 1080) / 54000 = 2273/54000 = 0.04209. Then SE = sqrt(p̂(1 − p̂)(1/27000 + 1/27000)) = sqrt(0.04209 × 0.95791 × 7.4074×10⁻⁵) = 0.001728. The observed difference is 0.04419 − 0.04000 = 0.00419, so z = 2.42 and the two-sided p is 0.0155 - the dashboard is arithmetically correct.
The 95% interval for the absolute lift is 0.42 pp ± 1.96 × 0.173 pp, i.e. [0.08 pp, 0.76 pp]; in relative terms a 2% to 19% lift. The point estimate is real-looking but the interval is wide, and the bottom of it is not obviously worth a redesign plus its maintenance cost.
The 0.015 figure is only valid for a single, pre-specified look at the planned sample size. The filed design says 14 days; this is look 9 of 14, so the statistic does not have the reference distribution the p-value assumes. Under the design as filed the answer is no - the test has not completed, and the number quoted is a random-time statistic, not a confirmatory one.
2. Quantify the multiplicity introduced by nine interim looks and six metrics. What decision boundary would you actually need, and what does that imply for day 9?
Read the worked answer
If every metric is tested at every look there are up to 9 × 6 = 54 tests, so the chance of at least one p < 0.05 under the null is roughly 1 − 0.95⁵⁴ ≈ 94%. Even restricting to the primary metric, nine looks give 1 − 0.95⁹ ≈ 37% false-positive risk.
A crude Bonferroni control sets α* = 0.05/54 = 9.3×10⁻⁴, which needs |z| ≥ 3.32 two-sided. The observed 2.42 is nowhere near. Bonferroni is too conservative here because the looks are highly correlated (the same accumulating users) and the metrics are correlated, so it over-penalizes.
The appropriate tool is a group-sequential boundary with alpha spending. With O'Brien–Fleming spending across 14 planned looks, at information fraction 9/14 = 0.64 the two-sided boundary sits near |z| ≈ 2.5 (nominal p ≈ 0.012). The day-9 result at z = 2.42 is therefore borderline at best, and any adjustment for the six secondary metrics pushes it clearly inside the null region.
Implication: the ship decision cannot rest on this read-out. Either wait for the pre-registered horizon, or pre-specify a single primary metric and a spending function and re-run.
3. Evaluate the post-day-4 exclusion of low-pageview sessions and the newsletter sign-up result. What conditions would you require before shipping?
Read the worked answer
The exclusion is the more serious problem. Randomization assigns users to a header; page views within the session are a post-treatment variable that the header plausibly affects (a header that invites clicks changes how many articles a user opens). Conditioning on it breaks intent-to-treat: if the new header reduces low-pageview sessions, they are dropped disproportionately from the new arm, the denominator shrinks, and CTR is mechanically inflated. The exclusion rule was also introduced after data existed, which makes it an analyst degree of freedom. The fix is to analyze all randomized users, report the excluded fraction per arm, and treat pageview-based cuts as a pre-specified subgroup if wanted at all.
The newsletter result is a textbook multiplicity artifact: with six correlated metrics, P(at least one p ≤ 0.05) ≈ 26% under the null, so a single p = 0.03 is unremarkable; it also has no pre-specified direction.
Conditions to ship: intent-to-treat re-analysis on the full sample; one pre-registered primary metric with a sequential boundary; a sample-ratio-mismatch check confirming the 50/50 split held after filtering; and a decision rule agreed in advance, e.g. ship only if the lower 95% bound on absolute lift exceeds the 0.15 pp threshold the team says is the minimum worthwhile effect.