The A/B Test That Beat the Control by 0.4 Points
A checkout redesign was tested for two weeks. Control had 48,200 sessions and a 3.1% conversion rate; treatment had 47,900 sessions and a 3.5% conversion rate. The product manager wants to ship. A analyst notes the p-value is 0.04. During the test, a payment provider outage on day 9 lasted four hours and affected both arms, but traffic routing was not randomized at the session level: users were assigned by cookie, and returning users kept their original assignment. Mobile traffic was 62% of sessions and converted at 2.4% overall versus 4.9% on desktop. The team also peeked at results on days 4, 7, 10, and 14.
1. Assess the strength of the evidence and identify the specific threats to validity in this test.
Read the worked answer
A 3.1% to 3.5% lift on roughly 48,000 sessions per arm is a real-looking difference, and p = 0.04 is nominally significant at 0.05. But several things weaken it.
First, peeking. Checking at days 4, 7, 10, and 14 and stopping when significant inflates the false-positive rate well above 5%. A p-value read after four looks is not the same as a p-value from a single pre-planned analysis.
Second, assignment by cookie. Returning users keep their assignment, which is fine for consistency but means the arms are not independent random samples of sessions; the effective sample size is closer to the number of users, and any systematic difference in which users return could bias the result.
Third, the outage on day 9. If it hit both arms equally it adds noise rather than bias, but four hours is small relative to two weeks, so it is a minor concern.
Fourth, the mobile/desktop mix. If assignment was not stratified and the arms differ in device mix, the comparison is confounded. With 62% mobile and a large desktop/mobile gap, even a small imbalance matters.
2. What would you recommend, and what analysis would you run before deciding?
Read the worked answer
I would not ship on this evidence alone. I would run a segmented analysis first: conversion by device within each arm, and a check that the mobile/desktop proportions match across arms. If the lift is concentrated in one device segment with a small sample, that is a different and weaker claim than a uniform lift.
I would also recompute the test using users, not sessions, as the unit, since assignment was by cookie. And I would apply a correction for the four peeks, or simply treat the day-14 result as one of several looks and require a stricter threshold.
If the segmented analysis shows a consistent lift across devices and the user-level test holds, I would recommend a confirmatory run with pre-registered duration, session-level randomization, and device stratification. If the lift vanishes in one segment or the user-level test is not significant, I would not ship. The cost of a second two-week test is small against a checkout change that affects all traffic.