When the Sequential Test Won't Stop Winning
A SaaS company tests a new pricing-page layout (Experience B) against the current layout (Control A) with a primary metric of trial signups. The team checks results daily and plans to stop as soon as confidence exceeds 95%. On day 3, B shows +6% at 96% confidence; the team prepares to ship. On day 5, B shows +1% at 71% confidence. On day 9, B shows +7% at 97%. Traffic is 40k sessions/week, and signup rate is about 2.5%. The analyst notes the daily swings and asks whether the test is broken. The team has no pre-registered sample size and no fixed end date.
1. Explain what is happening statistically and why the team's stop rule is unsafe.
Read the worked answer
The team is peeking repeatedly and stopping on a threshold, which inflates the false-positive rate far above 5%. Each daily look is another chance for random fluctuation to cross 95%; with many looks, a test with no true effect will appear significant at least once with high probability. The day-3 "win" and day-9 "win" are consistent with noise around a small or zero effect, and the day-5 dip is the same noise.
Compounding factors:
- Low baseline rate (2.5%) means conversions accumulate slowly; 40k sessions/week yields roughly 1,000 signups/week, so early reads are underpowered and volatile.
- No pre-registered sample size or end date means there is no principled stopping point - the team can always wait for a favorable day.
The test is not "broken"; the decision process is.
2. Recommend a corrected design and a decision rule the team can commit to before relaunching.
Read the worked answer
Corrected design:
- Pre-register the primary metric, minimum detectable effect (e.g., +5% relative), alpha (0.05), power (0.80), and required sample - then compute the run length from the 2.5% baseline and 40k/week traffic.
- Choose one valid stopping approach: a fixed-horizon test (analyze only at the pre-set end) or a sequential/always-valid method designed for continuous monitoring. Do not mix daily peeking with a fixed 95% rule.
- Run at least one full business cycle (typically 2+ weeks here) to cover weekly seasonality.
Decision rule to commit in writing:
- If the pre-set end is reached and the interval excludes zero in favor of B, ship B.
- If it excludes zero against B, keep A.
- If it includes zero, call it inconclusive and either extend once by a pre-agreed amount or move on - never re-run until it "wins."
Also fix the readout cadence: check for health (traffic, QA) daily but do not evaluate significance until the pre-set point.
3. The stakeholder says "but we saw 97% on day 9 - that's real." How do you respond?
Read the worked answer
Acknowledge the number is real as a data point but explain it is not a valid basis for a decision because it was selected after many looks. The probability that some day crosses 95% under no true effect is high; picking the best day is cherry-picking. The honest summary is that the observed effect is unstable (ranging from +1% to +7%), which is what you expect when the true effect is small relative to noise at this sample size. Offer the pre-registered plan as the path to a defensible answer, and note that a stable, real +5% effect would not swing from 71% to 97% confidence in four days at this traffic.