The checkout test that nearly passed
An online retailer tested a redesigned checkout. Over 14 days the control converted at 3.10% (372 of 12,000 sessions) and the variant at 3.40% (408 of 12,000). The analytics team reports a two-sided p-value of 0.08 and a 95% confidence interval on the lift of −0.03 to +0.63 percentage points. The test received roughly twice the traffic originally planned, so the growth manager wants to 'call it' and ship. Two things are known. The variant adds an extra step for returning customers. And the traffic split shows 30% mobile in the test against a 55% mobile share in normal traffic. Average order value is €64 at a 32% gross margin.
1. Interpret the p-value and the confidence interval. Can you conclude the variant does not work?
Read the worked answer
What p = 0.08 says. If the variant had no true effect, there is an 8% chance of seeing a difference at least this large. That is above the conventional 5% threshold, so the result is not statistically significant - it fails to clear the bar set in advance.
What it does not say. It does not say the variant has no effect. The 95% interval runs from −0.03pp (a rounding-error loss) to +0.63pp (a large gain). The data are inconclusive, not negative. 'Not significant' and 'no effect' are different sentences, and conflating them is how organisations ship noise and kill real improvements.
What the effect would be worth if real. At the point estimate of +0.30pp:
- Gross profit per session = 0.0030 × €64 × 32% = €0.061
- At 100,000 sessions a month: ~€6,100 per month, ~€74,000 a year
That is large enough to justify another two weeks of testing and too large to dismiss on a p-value of 0.08.
The power problem. The test was underpowered for the effect it was trying to detect. At a baseline of ~3.25% and a 0.30pp minimum detectable effect, the required sample is roughly n = 2(1.96 + 0.84)² × 0.0325 × 0.9675 ÷ 0.003² ≈ 55,000 sessions per arm. They ran 12,000. The test could not have reliably detected the effect it was designed to find.
2. What does the mobile-share discrepancy (30% vs 55%) do to the result?
Read the worked answer
It potentially invalidates the estimate entirely - this is more serious than the p-value.
It is a sample ratio mismatch. Random assignment should make the test population's device mix match live traffic. 30% versus 55% means one of two things, and both are fatal to interpretation:
- A tracking or assignment bug. Mobile sessions were dropped, filtered, or failed to randomise - for example, a redirect that breaks on some mobile browsers. If assignment is correlated with device, the arms are no longer exchangeable, and the comparison is between two differently-composed groups, not two versions of the page.
- A genuine population shift (a campaign, an app deep-link, a seasonality effect). Then the result measures an effect on a population that is not your customer base - and mobile converters typically behave very differently from desktop.
Diagnostics, in order:
- Chi-square test on the assignment counts by device. A ratio mismatch is a red flag before any outcome is examined.
- Segment conversion by device in both arms. If desktop alone shows a clean, significant lift, that is a usable (if narrower) finding.
- Audit the event pipeline for the two weeks and confirm the mobile share in the raw server logs, not the analytics tool.
Decision consequence. Do not ship on this data. Whatever the +0.30pp represents, it is not an estimate of the effect on live traffic - 55% of which is mobile and under-represented by nearly half in the experiment.
3. Recommend a plan for the next 30 days.
Read the worked answer
Do not ship to 100%.
- Freeze the decision and fix instrumentation first (days 1–5). Resolve the device-allocation bug and confirm that assignment is random and logged at the server. Any result produced before this is fixed is uninterpretable.
- Pre-register the next test (days 1–2): primary metric, minimum detectable effect, sample size (~55k per arm for 0.3pp), duration (minimum two full weeks, covering at least one payday and one weekend cycle), and a stopping rule. The current test was analysed at 2× the planned traffic - that is optional stopping, and it inflates false positives.
- Change the primary metric to revenue per session, not conversion. The variant adds a step for returning customers. Conversion can rise while repeat purchasing falls, and conversion alone will not reveal it. Revenue per session captures both.
- Run the retest (days 5–26) with a device-stratified read-out. Pre-specify that if the desktop-only effect is significant and the mobile effect is neutral, a desktop-only rollout is an acceptable interim outcome.
- Interim mitigation. If the qualitative research supports the redesign, ship it to desktop only - a decision that is reversible, capped in exposure, and does not consume the credibility of the test programme.
What would justify shipping now: a clean ratio check, a pre-registered analysis, and a significant result on revenue per session. None of the three is currently satisfied.