Shipping a Borderline A/B Test
A product team runs an A/B test on a new checkout button. Control: 6,000 users, 4.6% conversion. Treatment: 6,000 users, 5.1% conversion. The two-sided p-value is 0.08. Revenue per conversion is $42; the new button adds one engineering day of maintenance per month. A subgroup analysis shows the effect is +1.2 points on mobile (n=2,100) and -0.1 points on desktop (n=3,900). The test ran for 9 days, including a holiday weekend. One guardrail metric, checkout error rate, rose from 0.9% to 1.4% in treatment. The team lead wants to ship because '5.1 is bigger than 4.6.'
1. Should the team ship, iterate, or stop? Give a decision and the three most important pieces of evidence.
Read the worked answer
Decision: iterate, do not ship yet.
- Statistical uncertainty: A two-sided p-value of 0.08 does not clear a conventional 0.05 threshold, and the absolute difference is 0.5 points. That is compatible with a true effect near zero.
- Guardrail regression: Checkout error rate rose from 0.9% to 1.4%-a 56% relative increase. A conversion lift that comes with more errors may not survive real traffic or may create support costs.
- Heterogeneity: The mobile effect (+1.2 points, n=2,100) and desktop effect (-0.1, n=3,900) point in different directions. That pattern may be real or may be noise from an underpowered subgroup, but it argues against a blanket rollout.
Stopping entirely would be premature because the point estimate is positive and mobile is promising. The next step is a focused test that fixes the error-rate issue and pre-specifies the mobile segment.
2. How would you redesign the next test to resolve the mobile/desktop ambiguity?
Read the worked answer
Redesign around pre-specification and power, not more slicing of the same data.
- Pre-register mobile and desktop as the primary segments, with a hypothesis: the new button reduces friction on small screens. Treat any other cut as exploratory.
- Stratify randomization by device and by traffic source so each segment has balanced groups.
- Power the test for the smaller segment. With 2,100 mobile users, the current test is underpowered for a 1-point effect; estimate the required sample and run until it is met.
- Run full weeks and avoid launching over a holiday weekend, which can change device mix and purchase intent.
- Add guardrails for checkout errors and support contacts, with a pre-agreed stop rule if error rate exceeds control by more than, say, 0.2 points.
- Fix the error cause first; otherwise the next test confounds button design with reliability.
3. The team lead argues that a 0.5-point lift is worth $X. Compute a rough annual value if traffic is 200,000 sessions/month, and explain why that number should not decide the question alone.
Read the worked answer
Rough calculation: 0.5% of 200,000 sessions = 1,000 additional conversions per month. At $42 revenue per conversion, that is $42,000 per month, or about $504,000 per year (ignoring compounding and seasonality).
That number is large enough to justify more testing, but it should not decide the question alone for four reasons:
- Uncertainty: The p-value is 0.08; the true lift could be zero or negative. A point estimate is not a guarantee.
- Guardrail cost: The error-rate increase may create refunds, support load, or churn that offsets revenue.
- Segment mix: If the lift is concentrated on mobile, the value depends on mobile traffic share and whether the effect persists.
- Maintenance: One engineering day per month has an opportunity cost, and a fragile button may slow future work.
Use the $504k as an upper-bound-sized reason to invest in a better test, not as a shipping authorization.