The A/B Test That Passed at Day Nine
An online subscription service tests a redesigned checkout page. Assignment is by hash of user ID; the plan was a 50/50 split over 14 days with 100,000 users per arm to detect a 3% relative lift in conversion at 80% power. At day 9 the growth team checks the dashboard: control 51,240 users at 4.10% conversion, treatment 49,880 users at 4.27%, p = 0.04. A cell chi-square on the assignment counts gives p < 0.001. Logs show that at day 5 an analytics release began dropping events from Android WebViews when a session exceeded 30 minutes, and that the treatment page is 1.8 seconds slower at the median on mobile. Two of twelve pre-defined segments show lifts above 8%. The VP of Growth wants to ship Monday and has asked for a one-paragraph justification to attach to the launch record.
1. Should the feature ship? Give the reasoning the launch record needs.
Read the worked answer
No. Three independent defects, any one of which is disqualifying. First, the sample-ratio mismatch: 51,240 versus 49,880 against a hash-based 50/50 assignment should be impossible unless exposure or logging is broken. When the denominator is wrong, the numerator - the conversion rate - cannot be trusted, and the direction of the bias is unknown. Second, peeking: the team checked daily and stopped at day 9 of a planned 14, so an unadjusted alpha has been spent repeatedly and a single p = 0.04 at an arbitrary look is close to noise. Third, the differential instrumentation bug: if WebView event drops trigger on long sessions, and the treatment page is 1.8 seconds slower, the bug hits arms unequally and can mechanically move the conversion rate in either direction.
Two of twelve segments above 8% is roughly what chance supplies from twelve looks. The honest launch note says the experiment is invalid, not that the result is negative.
2. Before rerunning, how do you determine whether the mismatch came from assignment or from logging?
Read the worked answer
Separate the two empirically before touching the results. The hash is deterministic, so re-derive the bucket for a sample of user IDs from the raw identifier and compare against the stored assignment. If recomputation matches, assignment was correct and the mismatch lives in exposure or event logging.
Then look for a step change rather than drift: compare daily counts per arm across the pre-period. A discontinuity aligns with a deploy, and the day-5 analytics release is the obvious suspect - check for a deduplication or identity-stitching change, which produces exactly this signature. Confirm the ratio also holds in a window before the experiment began; if it does not, the pipeline was already broken. If logging is the culprit, rebuild the metric from raw events in both arms and rerun the cell chi-square on corrected denominators. Only a test that passes the SRM check at a pre-registered sample size is read at all.
3. Rewrite the experiment design so the next run is trustworthy.
Read the worked answer
One primary metric plus a short pre-registered set of guardrails - latency, error rate, revenue per user. Fix the horizon and sample size in advance, and if the business genuinely needs interim looks, use group-sequential boundaries or always-valid confidence sequences so peeking is priced into the alpha rather than forbidden by memo.
Instrument exposures server-side with a stable ID, and gate every experiment on an A/A test that must pass the sample-ratio check. Run the analytics release as its own controlled change, never mid-experiment. Where latency is a plausible mechanism for the outcome, either fix the 1.8 seconds first or pre-register a latency-adjusted analysis. Treat segments as exploratory and confirm any segment finding in a dedicated follow-up with its own power calculation. Report the effect as an interval against the 3% threshold rather than a p-value, and archive the pre-registration so the next day-nine dashboard glance cannot quietly become the decision.