A/B Test Shows Open-Rate Lift but No Revenue Lift
A B2B software company sells a $99/month tool. It ran a subject-line A/B test on a monthly newsletter to 10,000 contacts, split evenly. Variant A: 'Your September product update.' Variant B: '3 new ways to automate reporting.' Results: A opened 24.1%, B opened 31.2%; clicks were 1.9% and 2.0%; demo requests attributed within 7 days were 14 and 12. The marketing manager wants to roll out B to the full 40,000-contact list and declare a 29% improvement. The company uses Apple Mail for about 35% of recipients and has no holdout. Last quarter, a similar 'winning' subject line produced no revenue difference. The team has a HubSpot A/B test tool but no statistical significance calculation configured.
1. Interpret the test results. Should the team roll out variant B? What cautions apply?
Read the worked answer
No-do not roll out B on the claim of a 29% improvement. The open-rate gap is large, but opens are an unreliable metric when about 35% of recipients use Apple Mail, because Mail Privacy Protection can prefetch pixels and inflate opens unevenly. The click rates are essentially flat (1.9% vs 2.0%), and demo requests actually favor A (14 vs 12), though the numbers are too small to be conclusive. With roughly 5,000 per variant, a difference of two demo requests is well within noise. The prior 'winning' subject line that produced no revenue difference is a warning against repeating the same mistake. Subject lines can change open behavior without changing business outcomes. Treat B as a hypothesis, not a winner, and investigate whether the open lift is real or an artifact. Rolling out to 40,000 contacts would multiply an unproven change and make the result harder to reverse.
2. Design a follow-up experiment with a primary metric, sample size considerations, and a holdout to determine whether the subject-line approach drives business results.
Read the worked answer
Set the primary metric before the test: demo requests or qualified leads per 1,000 delivered, with revenue as a secondary lagging metric. Use the full 40,000-contact list, randomize at the contact level, and hold back 10% as a no-send control for the campaign period to separate email effects from seasonality. Run for at least four to six weeks or enough sends to reach a precomputed sample size for a minimum detectable effect; do not stop early on a promising open rate. Keep creative and offer constant and test only the subject-line approach. Report per-variant delivered, clicks, demo requests, and unsubscribes, with confidence intervals. Check segment-level results by mail client and lifecycle stage. If B does not beat A on the primary metric, keep the simpler subject line and test a different lever, such as offer or send time. The holdout is essential because it shows whether the newsletter is causing demos or merely correlating with a good month.