Feature Experimentation and Feature Management
What the exam tests
This domain blends experimentation with feature management. An experiment assigns users to variants to measure causal impact; a feature flag controls rollout independently of code deploys. Know the experiment lifecycle: hypothesis, exposure event, variant assignment, metric selection, and results interpretation (statistical significance, sample size, guardrail metrics).
Key mechanics
- Exposure events mark when a user actually sees a variant; without exposure, analysis is biased.
- Assignment should be deterministic and sticky (a user stays in one variant).
- Feature flags enable gradual rollouts, targeting, and kill switches.
- Metrics: primary success metric plus guardrails to catch regressions.
Traps to watch
- Analyzing users who were assigned but never exposed (dilutes effect).
- Peeking at results early and stopping on noise - significance requires adequate sample size.
- Confusing a feature flag (rollout control) with an experiment (causal measurement).
- Ignoring guardrail metrics that reveal harm even when the primary metric rises.
Rule of thumb
No exposure, no experiment - only analyze users who triggered the exposure event. Worked example: to test a new checkout button, define checkout_viewed as the exposure event, assign 50/50 sticky variants, track purchase_completed as primary and page_error as a guardrail, and wait for the precomputed sample size before deciding.