Day Nine of a Checkout Experiment
A subscription business is testing a redesigned checkout. The pre-test power calculation called for 148,000 users per arm to detect a 5% relative lift in conversion from a 3.2% baseline, with 80% power at alpha 0.05 two-sided. On day nine, with 61,000 users per arm, conversion is up 4.6% relative and the p-value is 0.048. The product manager wants to ship on Monday. A guardrail metric, revenue per user, is down 2.1% with p = 0.19. A post-hoc breakdown by device shows +9% on mobile and -3% on desktop, and the team also sliced by six acquisition channels; mobile is the only slice below 0.05. On day three an anti-bot filter was deployed that removed 8% of sessions, at different rates in the two arms.
1. Should the feature ship? Reason quantitatively about the strength of the evidence.
Read the worked answer
No - the evidence is not there yet.
The test has reached 61,000 of a planned 148,000 users per arm, about 41% of the information. At that sample size the minimum detectable effect is far larger than 5%: using the standard approximation for a two-proportion test, the detectable relative lift is roughly 8 to 9%. A 4.6% observed lift sits inside the noise band the design was never built to resolve, so the point estimate is unstable and its interval plausibly runs from near zero to well above 9%.
A p-value of 0.048 is also not the nominal 5% after repeated looks at accumulating data. Monitoring daily and stopping when the number first dips below 0.05 inflates the false positive rate substantially - with frequent looks it can exceed 20%. A valid early stop needs a sequential design; an O'Brien-Fleming spending boundary at this information fraction would demand roughly p below 0.005.
On the guardrail, p = 0.19 is absence of evidence, not evidence of safety: the test was powered for conversion, not revenue. The -2.1% direction is a warning. Recommendation: continue to plan or switch to a proper sequential test.
2. How should the device and channel findings be interpreted and reported?
Read the worked answer
Treat them as hypothesis-generating, not as findings. One primary metric plus six channel slices plus two device slices means at least nine comparisons; at alpha 0.05 the chance of at least one spurious 'significant' result is 1 - 0.95^9, roughly 37%. Mobile being the sole slice below 0.05 is exactly the pattern that random noise produces.
The correct test for 'does the effect differ by device' is a formal interaction between variant and device, reported as a difference-in-differences with a confidence interval - not two separate p-values compared by eye. If the team wants channel-level claims, pre-register one or two segments with a stated rationale and apply a corrected threshold; Bonferroni over six channels would require about p < 0.008, which mobile will not survive.
The day-three bot filter is a second problem: it removed 8% of sessions at uneven rates across arms, which can manufacture or erase segment effects. Re-analyse on an intent-to-treat basis using all assigned users, with the filtered population as a sensitivity check, and confirm that assignment itself still shows no sample ratio mismatch. Any segment signal worth acting on should be replicated in a confirmatory test.
3. What changes should the organisation make to its experimentation programme before the next test?
Read the worked answer
Move from fixed-horizon, peek-driven decisions to a design that permits monitoring: either commit to the planned sample and forbid interim reads, or adopt a sequential method with a pre-specified alpha-spending function, which is what the daily dashboard actually implies.
Pre-register the primary metric, one or two segments with a stated hypothesis, and guardrail non-inferiority margins. The revenue-per-user guardrail should have a margin - for example, no more than 1% degradation - rather than being judged by whether p crosses 0.05.
Forbid instrumentation changes mid-flight; if one is unavoidable, treat the pre-change period as a separate analysis and report both. Add a sample ratio mismatch check that runs daily to catch assignment or filtering faults within hours.
Reduce the sample burden with variance reduction (CUPED using pre-period behaviour) so that a 5% effect becomes detectable at a fraction of 148,000 per arm. Finally, separate the decision-maker from the analyst in the readout: the person who wants to ship should not be the person choosing which cut of the data to believe.