Shipping a Mobile-Only Checkout Variant After a Noisy A/B Test
An online retailer ran a 14-day A/B test on its checkout page. Traffic was split 50/50 by user ID. Control had 60,000 sessions and a 3.10% purchase rate; the variant had 58,100 sessions and 3.42%. On day 7 the difference was significant at p=0.04; on day 14 p=0.11. A chi-square check on assignment counts gives p=0.01. The variant's effect is +18% on mobile and −2% on desktop; mobile is 70% of traffic. The growth team wants to ship the variant to mobile only, citing the day-7 result and the mobile lift. The analytics manager notes that the test was stopped early for a 'peek' and that the assignment imbalance may come from a bot filter applied after randomization. The company ships changes weekly and has no sequential-testing policy. The VP asks: can we ship mobile, and what should we do next?
1. Interpret the evidence: what do the sample-ratio mismatch and the day-7 versus day-14 p-values imply about the test result?
Read the worked answer
The SRM p=0.01 means the 60,000 vs 58,100 split is unlikely under 50/50 random assignment. A post-randomization bot filter can remove users differentially by arm, so the remaining groups are no longer comparable; the outcome comparison is biased until the cause is found. The day-7 p=0.04 is not confirmatory because the team peeked and would have stopped at the first significant result; repeated looks inflate the false-positive rate. The day-14 p=0.11 is consistent with no effect or an underpowered effect. The mobile +18% and desktop −2% result is a post-hoc subgroup; with 70% mobile, the overall relative effect would be roughly 0.7×18% + 0.3×(−2%) = 12%, but the observed overall lift is about 10%, so the subgroup story is plausible but not statistically established. Multiple comparisons and the SRM make the mobile-only ship decision unsupported.
2. What should the team do next if it wants a defensible ship/no-ship decision within four weeks?
Read the worked answer
Stop the test and audit the assignment and bot-filter logic; fix filtering so it happens before randomization or is pre-specified and applied identically. Re-run with a fixed horizon and sample size powered for a minimum detectable effect, or use a sequential test with alpha spending if early looks are required. Pre-register one primary metric and the mobile/desktop interaction as a secondary analysis. Check SRM daily with an alarm. Analyze with a time-based or CUPED adjustment if pre-period data exist, and report confidence intervals. Ship only if the overall effect is significant and practically meaningful; ship mobile-only only if the interaction test is significant and the decision was pre-specified. Four weeks is enough for a clean rerun if traffic is similar. If the rerun is inconclusive, do not ship; the cost of a false positive is higher than a delayed launch.