Checkout A/B Test: Significant Conversion, Flat Revenue
Lumen Outfitters ran a 50/50 A/B test of a redesigned checkout flow: 180,000 sessions over nine days, including a holiday weekend. Conversion was 2.10% in control and 2.36% in treatment, with a two-sided p-value of 0.04. The growth team stopped the test as soon as it crossed significance. Revenue per session, however, was $1.02 in control versus $1.01 in treatment, because the new flow reduces attach rate on add-on items such as gift wrap and express shipping. When you segment, new visitors show a 0.45 percentage-point conversion lift while returning visitors show a 0.12-point decline. The product manager wants to ship the redesign next week and cite the conversion result in the launch announcement. The test had no pre-registered primary metric and no sequential-testing correction.
1. Interpret the result. What does the test actually support and what does it not support?
Read the worked answer
Three things weaken the conversion result. First, the test was stopped when it crossed significance, which inflates the false-positive rate; the nominal p=0.04 is not trustworthy without a sequential correction. Second, revenue per session is flat - the conversion gain is offset by lower add-on attach, so the business outcome is ambiguous. Third, the segment split shows the lift is concentrated in new visitors, with a decline among returning visitors; a pooled average hides a possible harm to the higher-value segment.
The honest interpretation is: the redesign may increase conversion for new visitors, but there is no evidence of a revenue gain, and there is a signal of a small negative effect for returning visitors. The p-value answers "is there an association under the null?" not "should we ship?" The decision depends on strategy - if new-customer acquisition is the goal and add-on revenue is not, the result might still be useful - but that must be stated explicitly, not inferred from conversion alone.
I would also check sample ratio mismatch, whether the holiday weekend changed traffic mix, and whether the 0.12-point decline in returning visitors is within noise. At a 2.1% base rate, segment-level standard errors are large enough that the decline may be unreliable.
2. Do you ship next week? What confirmatory work would you require?
Read the worked answer
I would not ship next week on this evidence. I would require a confirmatory test with a pre-registered primary metric, a fixed sample size or a proper sequential design, and at least two full weeks to cover weekday and weekend variation. The primary metric should be revenue or contribution margin per session, with conversion as a secondary; otherwise the team will keep optimizing a metric that does not pay.
Before the retest, diagnose the add-on decline: did the new flow remove a prompt, change page order, or add cognitive load? If the add-on loss is fixable, a revised design may capture both conversion and revenue. I would prespecify the new-visitor versus returning-visitor analysis and power it, rather than discovering it post hoc.
For the launch announcement, I would ask the PM to say: "A nine-day test showed a 0.26-point conversion increase with no revenue change; a confirmatory test is planned." Or delay the announcement. Citing p=0.04 from a stopped test as proof is a claim that will not survive a second look, and it trains the organization to over-read noise.
3. The PM argues that flat revenue is fine because conversion is a leading indicator. How do you respond?
Read the worked answer
A leading-indicator argument is only valid if there is evidence linking the intermediate metric to downstream value. Conversion can rise while order value falls, return rates rise, or new customers churn faster. Here the treatment already shows a lower attach rate, so the burden is on the PM to show that new visitors acquired through the new flow have equal or better lifetime value.
I would propose a cohort analysis: compare 90- and 180-day repeat purchase rate and revenue per acquired customer for new visitors in control versus treatment. If the cohorts are equivalent, conversion may be a legitimate leading indicator for that segment. If not, the conversion gain is a transfer, not a gain.
I would also ask what decision the leading indicator is meant to support. If the goal is new-customer acquisition, then the metric should be cost per acquired customer and early LTV, not raw conversion. If the goal is revenue, conversion is the wrong primary. The PM may be right that shipping is worthwhile, but the claim should be calibrated to the evidence, and the launch announcement should not overstate it.