Experiments
Experiments
Experiments tests whether you can design, run, and read Mixpanel A/B tests correctly. The exam covers hypothesis framing, exposure logging, variant assignment, statistical significance, and the operational details that invalidate results.
Core rules
- Define one primary metric before launch, plus guardrail metrics that must not regress. More metrics increase the chance of a false positive.
- Randomize at a consistent unit, usually user or account. Mixing units or changing assignment mid-test contaminates the comparison.
- Log an exposure event when the user actually sees the variant. Assigning a variant without exposure creates dilution and understates the effect.
- Sample size and power should be estimated in advance. A significant result on day two is not the same as a planned test duration.
- Read lift with a confidence interval, not only a p-value. Check for sample ratio mismatch between variants before trusting the result.
- Segment analysis is exploratory unless pre-registered; slicing many segments produces spurious winners.
Worked example
A paywall test compares control against a shorter trial. The primary metric is 14-day paid conversion; guardrails are refund rate and support contacts. Users are randomized at account level, and exposure fires when the paywall renders. After the planned period, variant B shows +3.1% conversion with a 95% interval of +0.4% to +5.8%. Because the interval excludes zero and no guardrail degraded, the result is credible. The reasoning: the interval shows plausible effect size, while the pre-set metric and exposure event prevent post-hoc storytelling.
Traps
Peeking repeatedly and stopping at the first significant result; changing the primary metric after seeing data; logging exposure after the user acts; ignoring sample ratio mismatch; testing many variants without correction; and attributing seasonal or novelty effects to the variant.