The A/B Test With a Confounded Result
A media company runs a campaign to drive app installs. They split the audience 50/50: variant A gets a push notification with a 7-day free trial message; variant B gets the same push with a 30-day free trial message. The campaign runs over a holiday weekend. Midway through, the marketing lead adds a 10% discount code to variant A only, because installs looked low. Final results: variant A 4.2% conversion, variant B 3.1%, and the team declares the 7-day message the winner and rolls it out company-wide. A week later, installs from the rollout are flat.
1. Identify every reason the original test result should not be trusted as evidence that the 7-day message is better.
Read the worked answer
Multiple confounds invalidate the conclusion:
- Mid-flight change to variant A. Adding a discount code to A mid-test changes the treatment. A is no longer "7-day message"; it is "7-day message + discount." Any difference could be caused by the discount, not the message.
- No control for the discount. The comparison is now confounded - you cannot attribute the lift to message length.
- Holiday weekend timing. Behavior over a holiday is atypical; results may not generalize to normal periods.
- No stated statistical test. 4.2% vs 3.1% is reported without sample size or significance. With modest samples, that gap may be noise.
- Single run, no replication. One campaign, one weekend - a fragile basis for a company-wide rollout.
The rollout going flat is consistent with the lift having come from the discount (or from noise), not the message.
2. Redesign the test so the message-length question can actually be answered.
Read the worked answer
Isolate one variable and hold everything else constant:
- Single variable. Test only message length (7-day vs 30-day). No discount, no creative differences, no send-time differences.
- Freeze the test. No mid-flight changes. If a change is needed, stop and restart - don't mutate a running variant.
- Avoid atypical windows or run long enough to span normal and atypical periods.
- Predefine success metric and sample size so you know when the result is meaningful; check significance before declaring a winner.
- If the discount is a real hypothesis, test it separately - e.g., a 2x2 (message length × discount) or a sequential test - so effects aren't entangled.
- Replicate before company-wide rollout.
Principle: a test answers one question only if only one thing varies. The original test changed two things and measured neither cleanly.