The Checkout Test With a Traffic Mix Problem
An online furniture retailer ran a two-week A/B test of its checkout. Variant A was the existing three-step checkout; Variant B was a new one-page checkout. Analytics reports B's overall conversion at 4.45% versus A at 4.25%, with p=0.04, and recommends rollout. The test was intended to split traffic 50/50. A tag-management error meant desktop users were more likely to receive B: 75% of B's 40,000 sessions were desktop, versus 55% of A's 40,000 sessions. By device, B underperformed: mobile conversion was 2.8% for B and 3.2% for A; desktop conversion was 5.0% for B and 5.1% for A. Average order value is £210 and gross margin is 38%. The product team can run one more test before peak season, which starts in five weeks. The CEO wants a decision this week.
1. What does the headline result show, and why is it unsafe? Recompute the overall conversion rates from the device mix and compare with the reported figures.
Read the worked answer
The reported 4.45% versus 4.25% is a weighted average, and the weights are broken. Recompute:
- A: 40,000 × 55% = 22,000 desktop sessions; 22,000 × 5.1% = 1,122 conversions. Mobile = 18,000 sessions; 18,000 × 3.2% = 576. Total = 1,698; 1,698 / 40,000 = 4.245%.
- B: 40,000 × 75% = 30,000 desktop sessions; 30,000 × 5.0% = 1,500. Mobile = 10,000; 10,000 × 2.8% = 280. Total = 1,780; 1,780 / 40,000 = 4.45%.
So the headline is arithmetically right but causally wrong. B is worse in both mobile and desktop; it wins overall only because it received a much larger share of desktop traffic, which converts much better. This is a Simpson's-paradox pattern caused by assignment bias. The p=0.04 is not trustworthy because randomisation was compromised. The test does not show that one-page checkout improves conversion; if anything, it suggests B may reduce conversion, especially on mobile.
2. What rollout decision would you make this week, and why?
Read the worked answer
Do not roll out B. The burden of proof is on the variant, and the only clean segment evidence points negative. Rolling out now would expose peak-season traffic to a likely mobile conversion loss. Keep A live. Fix the tag manager and assignment logic. If a decision is needed before peak, run a short corrected test with 50/50 stratified assignment by device and a pre-registered primary metric. Because mobile is a major traffic source, even a small mobile loss can outweigh desktop gains. The CEO should accept a short delay rather than scale a confounded result.
3. Redesign the test so that the next result is decision-grade. Specify randomisation, metrics and analysis rules.
Read the worked answer
Randomise at user level; stratify by device and new versus returning users; check sample-ratio mismatch daily. Pre-register the primary metric as revenue per session or conversion, with a minimum detectable effect and power calculation. Run at least one full week and include guardrails: checkout error rate, page load time, average order value and support contacts. Analyse mobile and desktop separately only if these segments are pre-specified; do not pick a winner from a subgroup after the fact. Use a holdout or switchback if implementation cannot randomise cleanly. Report confidence intervals, not just p-values, and include a pre-agreed stopping rule.