Sample Ratio Mismatch in a Trial-Length Test
Stillwater is a subscription meditation app with 900,000 monthly active users. The growth team runs a two-cell experiment on the paywall: control shows a 7-day free trial, treatment shows a 14-day free trial. Users are randomized 50/50 at signup by user ID. After 10 days, the experiment has 48,200 control users and 44,900 treatment users. The treatment paywall fails to render for users who already have an active trial from a previous campaign, and those users are silently excluded from the analysis. Primary metric is 30-day paid conversion; guardrails are refund rate and support contacts per 1,000 users. Early results show treatment trial starts up 6% and 30-day paid conversion down 3%, but most treatment users have not reached day 30. The PM wants to call the test early because trial starts are significant.
1. What is the sample ratio mismatch risk here, and what would you do before reading any metric?
Read the worked answer
The observed split is 48,200 versus 44,900, while a 50/50 assignment on 93,100 users implies about 46,550 per cell. The 1,650-user gap is far beyond chance; a chi-square check gives a very small p-value. That is a sample ratio mismatch, and it usually signals a logging, assignment, or exposure bug rather than a true effect. The silent exclusion of prior-trial users is especially dangerous because it is differential: the treatment paywall fails only for those users, so the exclusion removes people from one cell in a way tied to treatment. Before reading conversion, audit assignment logs, verify that every randomized user has an assignment record, and compare the count of assigned users to exposed users. Reconstruct the analysis as intent-to-treat and include everyone assigned, even if the paywall did not render. If the mismatch persists after fixing the logging, treat the experiment as invalid and restart.
2. How should the team handle the immature 30-day metric and the early trial-start lift?
Read the worked answer
Trial starts are an early leading indicator, not the decision metric. A 14-day trial mechanically increases trial starts and delays the point at which users can convert to paid, so the 30-day paid conversion is both immature and structurally lagged for treatment. Reading it now compares different amounts of elapsed risk. The team should wait for the cohort to mature, or use a pre-specified time-to-event analysis that accounts for censoring. If they must monitor, use sequential testing or a fixed interim analysis plan rather than repeated peeking. The decision rule should be: ship only if the mature primary metric improves and refund rate and support contacts do not worsen. If trial starts rise but paid conversion falls after maturity, the longer trial is attracting users who would not have paid, and the guardrails should confirm whether that is acceptable.