Testing and Optimization
What to know
This domain focuses on running valid tests and interpreting results. Understand A/B vs. MVT vs. automated testing (Auto-Target/Auto-Allocate), statistical significance, confidence level, lift, sample size, test duration (business cycles, novelty effects), and common validity threats (sample ratio mismatch, multiple comparisons, peeking). Know traffic allocation, experience distribution, and when to use Auto-Target (personalization by algorithm) versus Auto-Allocate (find the best experience).
Traps in questions
- Declaring a winner before reaching required sample size or during a novelty spike.
- Confusing Auto-Target (personalizes per visitor) with Auto-Allocate (shifts traffic to the best-performing experience).
- Ignoring sample ratio mismatch (uneven traffic) as a validity problem rather than a result.
- Treating a non-significant result as "no difference" - it may be underpowered.
Worked example / rule of thumb
A test runs 5 days with 40% lift and 85% confidence. Decision: do not declare a winner; extend duration to cover a full business cycle and reach the target confidence (commonly 90–95%) and sample size. Rule of thumb: decide the duration and sample size before launch; do not peek-and-stop. Use Auto-Allocate when you want to maximize conversions during the test; Auto-Target when you want per-visitor personalization.