The Stalled A/B Test Nobody Can Read
A retailer runs an A/B test on a product detail page. Control: 5,000 visitors, 250 conversions. Experience A: 5,000 visitors, 300 conversions. The activity was launched 9 days ago and is scheduled to run 21 days. On day 6, the merchandising team pushed a site-wide footer redesign to production, and the QA lead noticed that the same visitor can see the footer redesign but still gets the old page layout. Traffic is split 50/50 by the built-in random allocator. The business stakeholder wants to call the test for Experience A now because '300 beats 250.' The analyst also reports that mobile traffic is 62% of the audience and the target audience setting is set to 'All Visitors' with no device targeting.
1. The stakeholder wants to declare Experience A the winner on day 9. What is your recommendation, and what evidence would you present?
Read the worked answer
Recommend waiting, and explain the statistical and operational reasons.
- Lift is real but not yet decisive. Control 250/5,000 = 5.0%; A 300/5,000 = 6.0%. Relative lift is +20%, but the absolute difference is 1 percentage point. With roughly 5,000 per arm, the standard error of the difference between proportions is about sqrt(0.050.95/5000 + 0.060.94/5000) ≈ 0.0046, so the observed difference of 0.01 is only about 2 standard errors - suggestive, not a slam dunk. More importantly, the test is only 9 days into a 21-day window and has not covered a full weekly cycle of behavior.
- The footer change is a confound. A site-wide production change during the test means the environment is not stationary. If the footer redesign differentially affects mobile conversion, the comparison is contaminated.
- No device segmentation. With 62% mobile, an aggregate result can hide a segment where A loses. Segment by device before declaring a winner.
Present the confidence interval, the day-over-day conversion trend, and a device-segmented view. Recommend running to the planned end date (or at least through two full weekly cycles) and confirming the footer change is unrelated to the page under test.
2. The QA lead says a visitor can see the new footer but still get the old page layout. Is that a bug? Explain how Target would behave and what you would check.
Read the worked answer
Not necessarily a bug. Target allocates visitors to experiences per activity, and a production footer change deployed outside Target is independent of the activity's allocation. A visitor assigned to Control will keep seeing the control page layout while also seeing the new footer, because the footer is part of the site template, not the tested experience. That is expected behavior.
What to check:
- Is the footer change inside the activity? If the footer was changed directly in production and not via a Target experience, it applies to all visitors regardless of allocation. Confirm the footer is not part of the tested page fragment.
- Audience and targeting. Verify the activity's audience rules and any page-delivery or geo/device targeting that could cause a visitor to fall through to Control.
- Allocation consistency. Confirm the visitor's allocation is stable (cookie-based) and that the QA lead is not clearing cookies or using a different browser/profile between checks.
- QA mode. Use Target's QA/preview links to force each experience rather than relying on organic allocation.
If the footer was intended to be part of the test, then it was implemented incorrectly - it should have been built into the experience, not shipped globally.