The Experiment That Won on the Wrong Metric
You run a feature experiment on a paywall redesign. The primary metric is 7-day subscription conversion. The variant shows a +1.8% lift with a p-value of 0.04. The team is excited and wants to ship. But you notice: the experiment was configured with a 50/50 split, yet the exposure counts are 48,200 in control and 51,900 in variant. The variant also has a 3% higher rate of users hitting the paywall in the first place, because the redesign changed the entry screen. Secondary metrics show a small drop in 30-day retention that is not statistically significant. The experiment has run for 9 days.
1. What concerns do you raise about the validity of the result, and how do you investigate each?
Read the worked answer
- Sample ratio mismatch (SRM). A 50/50 split with 48,200 vs 51,900 exposures is a meaningful imbalance. Run an SRM check; a chi-square on those counts is far from 50/50. SRM usually means a bug in assignment or exposure logging, and it invalidates the comparison. Investigate: does the assignment key differ between groups? Are exposures logged at different points for the two variants?
- The entry-screen change confounds the result. The variant drives 3% more users to the paywall. The +1.8% conversion lift may just be more people seeing the paywall, not a better paywall. Check conversion as a rate over exposed users, and separately over all eligible users, to separate the two effects.
- Peeking / duration. Nine days may not cover a full weekly cycle or the 7-day conversion window. Check whether the result is stable across days and whether the 7-day window has matured for recent exposures.
- Multiple comparisons. If many metrics were checked, a p=0.04 on one is weak evidence. Adjust or treat it as directional.
- Retention signal. A non-significant retention drop is not proof of harm, but combined with the above it argues for caution.
Recommendation: do not ship on this result; fix the SRM and re-run or re-analyze with the confound separated.
2. How would you redesign the experiment to answer the real question cleanly?
Read the worked answer
- Isolate one change. Either test the new entry screen alone or the new paywall alone, not both. If the product requires both, define the primary metric over all eligible users (intent-to-treat) so the entry-screen effect is included deliberately rather than accidentally.
- Fix assignment and exposure logging. Verify the randomization unit (user vs device) and that exposure fires at the same logical moment in both arms. Add an automated SRM check that halts analysis if the split deviates.
- Pre-register metrics. Primary: 7-day subscription conversion over eligible users. Guardrails: 30-day retention, refund rate. Decide the decision rule before launch.
- Run long enough. Cover at least two full weekly cycles and ensure the conversion window has matured before reading out.
- Power the test. Compute required sample size for the minimum detectable effect you care about; 100k users may be underpowered for a 1-2% effect on retention.
This makes the result interpretable and the ship/no-ship decision defensible.