Brightpath SaaS: Stopping an A/B Test at p = 0.04
Brightpath is a B2B SaaS company testing a new trial onboarding flow. The growth team ran an A/B test for three days and stopped when the result reached p = 0.04. Each arm had 1,842 trial sign-ups. The observed paid-conversion rates were 10.4% in the new flow and 8.7% in the control. The product team defines 'conversion' as any trial account that logs in during the 14-day trial; growth defines it as a trial that becomes a paying account within 30 days. The test report uses the product team's definition. Assignment was by browser cookie, and 14% of returning visitors cleared cookies and were reassigned. Mobile users were automatically routed to the old flow because the new flow was not responsive. The VP of growth wants to ship the new flow to all users on Monday and cite a 20% relative lift in the board deck.
1. Assess whether the 20% relative lift is a defensible estimate. Identify at least three threats to validity and explain how each could bias the result.
Read the worked answer
The 20% relative lift is not defensible. Three threats:
- Early stopping / peeking. The team stopped at p = 0.04 after three days. Repeatedly checking and stopping on the first significant result inflates the false-positive rate. The p-value is not adjusted for multiple looks, and three days is far shorter than the 30-day conversion window, so many trials have not had time to convert. The observed difference is likely to shrink.
- Cookie-based assignment and reassignment. 14% of returning visitors cleared cookies and could be assigned again, possibly to the other arm. This contaminates both arms and breaks the independence of assignment. It can bias the estimate in either direction and makes the stated sample size unreliable.
- Mobile exclusion / routing. Mobile users were automatically sent to the old flow. If mobile users differ in conversion propensity from desktop users, the test estimates an effect only for desktop-like traffic. Shipping to all users assumes the effect generalises, which is unsupported.
Also, with 1,842 per arm and an 8% baseline, the test is underpowered to detect a 1.7-point absolute difference reliably; the confidence interval is wide. A 20% relative lift is a point estimate, not a planning number.
2. The two teams disagree on the conversion metric. Explain how the choice of metric changes the decision and what should be agreed before shipping.
Read the worked answer
Metric choice changes the decision. The product team's 'logged in during trial' is an engagement proxy; it can rise while paid conversion falls. Growth's 'paid within 30 days' is the commercial outcome and aligns with revenue. If the report uses login conversion, the team may ship a flow that entertains users but does not sell. If it uses paid conversion, the sample needs the full 30-day window and enough events; 10.4% versus 8.7% on paid conversion is a smaller absolute count and likely not significant at this sample size.
Before shipping, agree one primary metric with a precise denominator (eligible trial sign-ups), a fixed outcome window (30 days), guardrails (refund rate, support contacts, revenue per account), and a decision rule. Secondary metrics can be reported but must not replace the primary. The definition should be written down and versioned so the board deck cannot switch metrics after the fact.
3. Recommend next steps: what to do with the test data, whether to ship, and what the board deck should say.
Read the worked answer
Do not ship to all users on Monday. Treat the three-day result as a promising signal, not evidence. Fix assignment: use a stable user ID or server-side assignment, not a cookie that can be cleared. Randomise mobile users too, or explicitly scope the claim to desktop and state that mobile is untested. Pre-register the primary metric, the 30-day window, the sample size, and the stopping rule. Re-run for a fixed duration and analyse with a method that accounts for sequential looks if interim monitoring is required.
The board deck should say: 'A short pilot showed a positive signal in desktop trial engagement; the paid-conversion estimate is preliminary, the test was stopped early, and a confirmatory test is running.' It should not cite a 20% lift as established. If a decision cannot wait, ship only to the tested desktop segment behind a holdback, with monitoring.