The Checkout Test That Won by Losing
A retailer tests a new checkout flow for 4 weeks. Assignment is a client-side JavaScript flag; on the mobile app the flag was suppressed for app versions below 5.2, which cover roughly 80% of app sessions. Results: control 210,000 sessions and 6,450 orders (3.1%); treatment 48,000 sessions and 1,945 orders (4.1%). Segment split - control: 86,100 desktop sessions with 4,477 orders, 123,900 mobile with 1,973 orders. Treatment: 37,440 desktop with 1,797 orders, 10,560 mobile with 148 orders. The product manager wants to ship the new flow to 100% of traffic on Monday. An analyst flagged an odd traffic split but it was never investigated.
1. The PM says treatment wins by a full percentage point. Compute the segment-level rates and give your verdict.
Read the worked answer
Compute within each device:
| Segment | Control | Treatment |
|---|---|---|
| Desktop | 4,477 / 86,100 = 5.20% | 1,797 / 37,440 = 4.80% |
| Mobile | 1,973 / 123,900 = 1.59% | 148 / 10,560 = 1.40% |
| Overall | 6,450 / 210,000 = 3.07% | 1,945 / 48,000 = 4.05% |
The treatment loses in both segments. Desktop drops 0.40 points, mobile drops 0.19 points, and the pooled number moves up a full point. That is Simpson's paradox: a weighted-average reversal.
Why: desktop converts roughly 3x better than mobile, and the device mix shifted violently. Treatment is 78% desktop (37,440 / 48,000); control is 41% desktop (86,100 / 210,000). The treatment arm is composed of the high-converting segment, so the pooled rate rises even though the flow is worse everywhere.
Verdict: do not ship. The pooled 4.1% is an artifact of composition, not a treatment effect. Any decision made on the aggregate number - ship, extend, reinvest - is built on a reversal.
Note also the weighting: the pooled figure implicitly claims desktop should be weighted 78/22. If desktop is genuinely a small share of real traffic, the pooled number is not just confounded, it is unrepresentative of the business.
2. What actually went wrong with the experiment, and what is the correct remediation before any re-run?
Read the worked answer
Root cause: the randomization was not independent of the outcome-relevant covariate.
- The client-side flag was suppressed for app versions below 5.2 (≈80% of app sessions), so device type partly determined assignment. This is a selection bias built into the assignment mechanism, not bad luck.
- It should have been caught by a sample ratio mismatch (SRM) check: observed 210,000 : 48,000 is 4.38 : 1 against an expected 1 : 1. Run chi-square on assignment counts daily and stop the test on a low p-value. This is a guardrail, not an analysis result - never 'fix' SRM by weighting.
Remediation:
- Do not ship, do not extend, do not re-weight into a decision. Post-stratification can estimate a device-weighted effect, but it cannot repair assignment that correlates with device; treat any such estimate as diagnostic only.
- Move assignment server-side, before the response is rendered, so no client-side version gate can interfere.
- Pre-register the primary metric, guardrails, segment cuts, and stopping rule. Multiple ad-hoc looks inflate false positives.
- Fix exposure logging - log assignment and exposure separately, so a session that never saw the flow is not counted as a failed conversion.
- Re-run and confirm SRM ≈ 1:1 before reading any result.
3. Traffic volume is 258,000 sessions over 4 weeks. If the true effect on conversion is around 0.1 absolute points, was this test ever powered to detect it?
Read the worked answer
No - and this matters even after the SRM is fixed.
Rough check: for a two-proportion test at α=0.05, 80% power, baseline 3.1%, a 0.1-point absolute lift (3.1% → 3.2%) needs on the order of hundreds of thousands of sessions per arm - roughly 500,000 per arm, so about 1M total. With 258,000 total, the test is underpowered by an order of magnitude for an effect that small.
Practical consequences:
- A non-significant result is not evidence of no effect. Report the confidence interval on the difference, not a bare p-value.
- If the business only cares about effects ≥0.5 points, run a power calculation up front and set the duration accordingly. Testing for four weeks and hoping is not a design.
- Underpowered tests plus optional stopping is the standard recipe for shipping noise. Combine low power with repeated peeking and you get a test that 'wins' regularly at random.
What to report to the PM: point estimate, CI, the SRM failure, and a recommendation to re-run with server-side assignment and a pre-computed sample size - not a one-line 'treatment won by 1 point.'