Reading an Underpowered, Peeked Checkout Test
An e-commerce team tested a simplified checkout (variant B) against the current flow (variant A), randomizing at the session level. After 9 days, each arm had 10,000 sessions. Conversion was 3.12% for A (312 orders) and 3.58% for B (358 orders); a two-sided test gave p = 0.07. The team had peeked at day 6, seen p < 0.05, and expected to stop. Mobile is 68% of traffic, and the mobile conversion gap between arms is larger than the desktop gap. Average order value is USD 84 with a standard deviation of USD 60. Building the change costs USD 120,000.
1. Interpret p = 0.07 in combination with the day-6 peek. What can and cannot be concluded?
Read the worked answer
Plain reading At the conventional 5% threshold, p = 0.07 is not statistically significant: the observed 46-order difference is compatible with no real effect. The point estimate favors B, but the data do not rule out zero.
The peek matters more than the p-value
- A single pre-planned look at p < 0.05 controls type I error at 5%. Repeated looks inflate it - with several checks, the chance of seeing a spurious p < 0.05 at least once can approach 15–20%.
- So the day-6 signal is weak evidence, and the fact that the result drifted back to 0.07 by day 9 is exactly what noise looks like.
- The team's mental model (we saw it, we should stop) is a stopping-rule violation, not a finding.
What cannot be concluded
- That B does not work. Non-significance is not evidence of no effect; the confidence interval around the difference plausibly includes both trivial and commercially meaningful gains.
- That B works. The estimate is directionally positive and underpowered.
What can be concluded The test as run cannot settle the question. The practical recommendation is to fix the design (pre-registered sample size and analysis date), not to declare a winner.
2. The mobile gap is larger than the desktop gap. How would you analyze that without over-claiming, given the sample size?
Read the worked answer
Why it is tempting and dangerous With 68% mobile, the mobile arm holds roughly 6,800 sessions per variant. A 3.1% vs 3.6% base rate means about 210 vs 245 orders - differences of a few dozen events. Slicing further (mobile by new/returning, by device OS) produces cells where a handful of orders drives the result. Any apparent subgroup win is likely noise.
Disciplined approach
- Pre-specify device as a single interaction test, not a fishing expedition: fit conversion on variant, device, and variant × device, and report the interaction p-value with its confidence interval.
- Report the interaction, not two separate tests. Two separate p-values invite cherry-picking the flattering one.
- Check for a mechanism before believing it. Does the simplified checkout plausibly help mobile more (fewer form fields, smaller viewport)? If there is a design rationale, the subgroup finding is more credible - but still needs confirmation.
- Treat it as a hypothesis for the next test, which should be stratified by device and sized for the mobile subgroup.
- Correct for multiple comparisons if any additional segments were examined.
Reporting language: the mobile effect is directionally larger but the study was not powered to detect an interaction; it should not be presented as a confirmed finding or used to justify a mobile-only rollout.
3. Given the USD 120,000 build cost, what decision do you recommend and what would you need to compute to defend it?
Read the worked answer
Recommendation: do not ship on this evidence; extend or re-run with a pre-registered design.
Why not ship
- The effect is not significant and the stopping rule was violated, so the estimate is unreliable in both directions.
- USD 120,000 is a sunk-cost decision only if the effect is real; at p = 0.07 you would be betting the build on a coin flip.
What to compute to defend the decision
- Minimum detectable effect. With 10,000 per arm and a 3.1% baseline, the test can reliably detect a lift of roughly 0.9–1.0 percentage points. The observed 0.46-point lift is below that. This is the core argument: the test was never capable of resolving an effect this size.
- Required sample. To detect a 0.46-point absolute lift (about 15% relative) at 80% power and 5% significance requires roughly 25,000–30,000 sessions per arm - about three times what was collected.
- Expected value at the observed effect. 46 extra orders per 10,000 sessions times USD 84 average order value ≈ USD 3,900 per 10,000 sessions in incremental revenue. Compare to the 120,000 build cost: you need roughly 300,000 sessions of traffic to break even on revenue (and less on margin). Whether that is weeks or years depends on traffic volume - get that number.
- Sensitivity. Recompute at the lower confidence bound of the effect. If break-even requires the point estimate to hold exactly, the investment is fragile.
Plan: run to a pre-registered 60,000 total sessions, single analysis at the end, decision rule agreed in advance (ship if the lower bound of the lift still clears break-even).