Audience Overlap Eats the Test
A subscription media company runs three concurrent Target activities on the article page: (1) a paywall-message test targeting "all visitors" at 30% allocation, (2) a newsletter-signup test targeting "returning visitors" at 50% allocation, and (3) a recommended-articles test targeting "mobile users" at 40% allocation. All three modify overlapping page regions and use the same success metric (subscription starts). After two weeks, the paywall test shows a 1.8% lift with a confidence interval that crosses zero, the newsletter test shows a 4% lift, and the recommendations test shows a 0.5% lift. The product owner wants to declare the newsletter test the winner and ship it. Traffic is 900k sessions/month; roughly 60% mobile and 35% returning.
1. Explain how audience overlap affects the validity of these results and why declaring the newsletter winner is risky.
Read the worked answer
The three activities overlap heavily: a returning mobile visitor can be in all three at once. Because all three modify overlapping page regions and share one success metric, their effects are entangled - the newsletter lift may partly reflect the paywall or recommendations experience the same visitor also saw, or the newsletter audience may be enriched with returning (more engaged) users who convert regardless.
Specific risks in declaring the newsletter winner:
- Confounded effect: its measured lift includes whatever the other two activities did to the same visitors.
- Selection skew: "returning visitors" is a behavioral segment correlated with intent, so its baseline differs from the paywall's all-visitor baseline; the 4% is not comparable to 1.8%.
- Mutual contamination inflates variance, which is consistent with the paywall's wide interval.
Shipping on a confounded 4% risks shipping a non-replicable effect.
2. Recommend a restructured test plan and specify how you would prioritize the three ideas.
Read the worked answer
Restructure so each visitor is exposed to at most one of the overlapping activities at a time, or sequence them.
Options:
- Collision groups / mutual exclusion: put the three activities in one exclusion group so a visitor qualifies for only one; this preserves clean per-activity reads at the cost of longer run time.
- Sequence: run the highest-priority test to completion, then the next, reusing the same metric definition.
- Separate pages/regions: if the offers truly touch different regions, isolate them so they cannot co-modify the same element.
Prioritization: rank by expected impact and traffic reach. The paywall test touches all visitors and the core monetization moment, so it likely warrants first or exclusive priority; the newsletter test has the strongest observed signal and a clear segment; the recommendations test is weakest. Use a simple prioritization framework (impact, confidence, effort) and document the order so the team does not re-litigate mid-flight.
3. What reporting caveats would you attach to the newsletter result even after restructuring?
Read the worked answer
- Report the segment definition alongside the lift; "returning visitors" results do not generalize to new visitors.
- Confirm the sample met a pre-agreed minimum and that the interval excludes zero under the restructured (non-overlapping) design.
- State the run duration covers at least one full weekly cycle to avoid day-of-week bias.
- Note that the metric (subscription starts) may lag; check for delayed conversions before calling it.
- Flag that a segment-level win should be validated on a fresh cohort before a full rollout.