Experimentation
What this domain tests
This domain covers how to design, run, and interpret experiments: hypotheses, sample size, significance, ablations, and benchmark hygiene for generative models. Questions often ask what a result does and does not prove.
Core rules and concepts
- State H0 and H1 before inspecting data. A p-value is P(data at least this extreme | H0), not P(H0 is true).
- Power depends on effect size, variance, alpha, and n. Underpowered tests produce noisy 'wins'.
- Confidence intervals communicate effect magnitude; a result can be statistically significant yet practically irrelevant.
- Multiple comparisons inflate false positives; correct with Bonferroni or Benjamini-Hochberg.
- Ablations isolate one change at a time. Change learning rate and data together and you learn nothing about either.
- Generative benchmarks drift: check contamination, prompt format, and decoding settings before comparing scores.
Worked example
An A/B test of two prompts: A succeeds 120/1,000 and B succeeds 150/1,000. The absolute lift is 3 percentage points, the relative lift 25%. Before declaring B better, ask whether the test had power to detect 3 points at the chosen alpha; with these counts the difference may be within noise. Also confirm the same model version, temperature, and evaluation rubric were used for both arms. If the B arm also changed temperature, the comparison measures a confound rather than the prompt.
Traps that cost marks
- Peeking repeatedly and stopping when p < 0.05.
- Treating p = 0.049 as a large effect.
- Reporting a single run as a stable result without variance across seeds.
- Comparing benchmark scores obtained with different prompt templates or decoding parameters.
- Confusing statistical significance with business or quality significance.