Borderline A/B Test and a Leaky Conversion Model
An online retailer tested a redesigned checkout for 14 days. Control had 8,100 users and 502 conversions; treatment had 8,050 users and 563 conversions. The primary metric was purchase conversion, pre-registered as a two-sided test at alpha 0.05. The calculated p-value was 0.041. A day-3 logging outage dropped 4% of treatment events but almost no control events; the analyst excluded incomplete sessions from both arms. The team also checked 10 secondary metrics; mobile conversion was significant at p=0.03, desktop at p=0.31. A data scientist proposes a logistic regression to predict conversion using age, device, prior purchases, and used_new_checkout, and reports AUC 0.86 on the same test data. The test ran during a normal sales period, and no sample-ratio mismatch was reported. Leadership wants to ship if the model confirms lift.
1. Interpret the A/B result. Should the team ship immediately? Address p-value, effect size, missingness, segmentation, and multiple comparisons.
Read the worked answer
Compute the rates: control 502/8,100 = 6.20%; treatment 563/8,050 = 6.99%; absolute difference = 0.80 percentage points, relative lift about 12.9%. With a standard error near 0.39 percentage points, the 95% CI is roughly 0.03 to 1.56 percentage points, so the p=0.041 result is borderline and the lower bound is near zero.
More importantly, the day-3 outage is differential: 4% of treatment events missing. Excluding incomplete sessions from both arms does not fix non-random missingness; if low-intent treatment users were dropped, the treatment rate is inflated. The team should run an intention-to-treat sensitivity analysis, compare missingness by arm and outcome, and check whether the outage could change the conclusion.
Ten secondary metrics at alpha 0.05 will produce about 0.5 false positives by chance; mobile p=0.03 is not convincing after Bonferroni (0.005) or FDR, and the mobile/desktop split needs an interaction test rather than two separate tests. Do not ship immediately; extend or replicate, and pre-specify one primary decision rule.
2. Evaluate the logistic regression proposal. Can it confirm the lift, and what would you change?
Read the worked answer
The logistic regression as described cannot confirm the lift. used_new_checkout is post-treatment: it is part of the causal pathway and will leak outcome information, inflating AUC. AUC 0.86 on the same test data is also optimistic because the model was evaluated on data used for choices; there is no separate validation or temporal holdout.
Use only pre-treatment features, split by time or user, and report calibration and lift over a baseline. If the goal is to estimate the treatment effect, fit a regression with a treatment indicator and pre-treatment covariates, not a model that predicts conversion using a post-treatment variable. A predictive model can support targeting, but it cannot replace the randomized comparison or repair the missing data. Report the A/B effect with its CI and the sensitivity analysis; use the model as a separate, clearly labeled exercise.