Checkout Rollout Reversal Across Device Strata
An e-commerce team tests a new checkout button by randomizing at the zip-code level: 40 zips get arm A and 40 get arm B. Because mobile-heavy urban zips were overrepresented in B, the device mix differs sharply. Arm A has 1,000 users: 100 mobile with 30 conversions, 900 desktop with 90 conversions. Arm B has 1,000 users: 900 mobile with 180 conversions, 100 desktop with 5 conversions. Overall, A converts 120/1,000 (12.0%) and B converts 185/1,000 (18.5%), so the team plans to ship B. In the Statistics tab, a colleague groups by device and finds A ahead on mobile (30.0% vs 20.0%) and ahead on desktop (10.0% vs 5.0%). The mobile promotion was not part of the randomized treatment. The team asks whether B is truly better and what to report to leadership.
1. Explain why the overall and device-specific comparisons disagree. What variable is responsible?
Read the worked answer
This is Simpson's paradox. The overall comparison pools two device groups with very different baseline conversion rates and very different weights across arms. Mobile converts much better than desktop in both arms (30% vs 10% for A, 20% vs 5% for B). Arm B is 90% mobile, while arm A is only 10% mobile, so B's overall rate is pulled up by the high-converting mobile group. Within each device stratum, A is better. Device mix, not treatment, explains the reversal. Because randomization was by zip code, device mix is not guaranteed to balance, so device is a confounder of the arm-outcome relationship. The unadjusted 18.5% vs 12.0% comparison is not a valid treatment effect.
2. Compute a device-adjusted comparison and state what you would report.
Read the worked answer
Use direct standardization. With a 50/50 mobile/desktop standard population, A's expected rate is 0.530% + 0.510% = 20.0%; B's is 0.520% + 0.55% = 12.5%. A is ahead by 7.5 percentage points. Using B's mix (90% mobile, 10% desktop), A would be 0.930% + 0.110% = 28.0%, versus B's observed 18.5%, a 9.5-point advantage. The stratum-specific differences are +10 points on mobile and +5 points on desktop. Report the adjusted effect and the stratum-specific results, not the pooled 18.5% vs 12.0%. Also report confidence intervals: the desktop B cell has only 5 conversions, so uncertainty is wide. Do not claim B is better.
3. What design or analysis change would you make before scaling?
Read the worked answer
Randomize within device, or stratify assignment by device and analyze with a regression that includes device as a covariate. Pre-register the primary metric and the analysis plan. Check for sample-ratio mismatch by device; the 100 vs 900 split is a red flag. Use intent-to-treat and cluster-robust standard errors because assignment was by zip. If the promotion cannot be controlled, wait for a period with stable traffic mix or analyze only within device. Before shipping, run a follow-up test powered for the smaller desktop stratum. Do not scale on the pooled result.