An A/B test that neither arm clearly wins
An online course platform tested a redesigned checkout page. Control: 12,400 visitors, 298 purchases (2.40%). Variant: 12,150 visitors, 261 purchases (2.15%). The analyst reports a p-value of 0.21 and recommends reverting. You notice the variant arm contains 49% mobile visitors against 40% in control, and mobile converts at roughly half the desktop rate in both arms. The test ran for 11 days with a 50/50 random split performed by a third-party tool. The product manager wants to keep the test running for another fortnight 'until it comes out significant', and the CEO asks whether the redesign should ship.
1. What can and cannot be concluded from a p-value of 0.21 on this result?
Read the worked answer
The p-value of 0.21 means that if the two pages truly converted identically, a gap at least this large would appear about 21% of the time by chance. That is not evidence of no effect - it is evidence of insufficient information to separate the effect from noise.
What you can say:
- The observed difference is 0.25 percentage points in favour of control (2.40% vs 2.15%).
- The test was underpowered for a difference this small. At a 2.4% base rate and roughly 12,000 per arm, the standard error of the difference is about 0.20pp, so the test had 80% power only for effects around 0.55pp or larger. A genuine 0.25pp effect would usually be missed.
What you cannot say:
- That the redesign is neutral, or that it is worse. Absence of significance is not equivalence - claiming 'no difference' would need a pre-specified equivalence margin.
- Anything about direction with confidence: the point estimate favours control, but the confidence interval comfortably includes zero and includes small positive effects for the variant too.
2. The mobile/desktop mix differs between arms. How does that affect the comparison, and what would you do about it?
Read the worked answer
A 49% versus 40% mobile split is too large for a working 50/50 randomiser and is itself a finding: check the assignment or tracking before trusting any conversion comparison.
If it is real, the imbalance biases the comparison against the variant. Illustrating with the stated assumption that mobile converts at half the desktop rate:
- Control: 0.60 × desktop rate + 0.40 × half that = 0.80 × desktop rate = 2.40%, so desktop ≈ 3.00% and mobile ≈ 1.50%.
- Variant: 0.51 × d + 0.49 × d/2 = 0.755d = 2.15%, so desktop ≈ 2.85% and mobile ≈ 1.42%.
- Re-weighting the variant to the control mix: 0.60 × 2.85 + 0.40 × 1.42 ≈ 2.28%, against 2.40%.
So the mix shift explains roughly half the apparent gap (0.25pp down to about 0.12pp) but does not reverse it. Action: report device-stratified rates rather than a single blended number, and confirm the assignment mechanism before drawing any conclusion at all.
3. How should you respond to the request to 'keep running until it comes out significant', and what should you tell the CEO?
Read the worked answer
On the request: repeatedly checking a test and stopping when it crosses a threshold inflates the false-positive rate well above the nominal 5% - you are effectively buying lottery tickets until one wins. If the team genuinely needs a longer run, fix the duration in advance (or use a sequential-testing method designed for continuous monitoring), and state the minimum detectable effect the business actually cares about before looking again.
On the evidence: the effect, if any, is about 0.25pp and the test could only reliably detect roughly 0.55pp. More traffic is the honest fix, sized in advance: detecting 0.25pp at a 2.4% base with 80% power needs roughly 100,000 visitors per arm.
To the CEO: do not ship on this evidence, and do not claim the redesign is worse. Say: the test was inconclusive; the point estimate favours the current page but the difference is smaller than the test could resolve; the arms also had an unexplained device-mix difference that needs fixing first. Then offer a decision: revert for now on risk grounds, or rerun with a corrected split and a pre-set sample size - and be clear which of those is being chosen and why. Also ask whether checkout conversion is the right success measure, or whether completion of first lesson matters more.