Experimentation
What to know
Good experiments start with a hypothesis, an explicit baseline, and controlled variables. Ablations change one factor at a time (for example, data mixture, LoRA rank, learning rate, or decoding settings) so effects are attributable.
Evaluation design decides whether results mean anything. Keep a held-out test set untouched, use a validation set for checkpoint selection, and check for leakage. Perplexity is a language-modeling metric and is only comparable under the same tokenizer and test corpus. Task metrics differ by task: exact match, token-level F1, pass@k for code (unbiased estimator over n samples with k successes), and generation metrics such as BLEU/ROUGE, which correlate weakly with human judgment.
LLM-as-judge is convenient but biased: position bias (prefers the first or second answer), verbosity/length bias, self-preference for the same model family, and rubric sensitivity. Mitigate by swapping order, pinning a rubric, and calibrating against human labels on a sample.
Statistics: report means over multiple seeds with confidence intervals (bootstrap is common), use paired comparisons on the same items, and correct for multiple comparisons when sweeping many configs. Random search beats grid search when only a few hyperparameters matter; early-stopping schedulers such as ASHA save compute.
Online A/B tests measure business metrics, and an offline win does not guarantee an online win.
Traps
- Quoting a single-seed gain as real improvement.
- Tuning on the test set or reusing it across many experiments.
- Comparing perplexity across tokenizers.
- Ignoring judge position and length bias.
- Multiple uncorrected comparisons producing false positives.
Worked example
If config A scores 71.2 and config B scores 71.8 on 500 items with three seeds each and overlapping confidence intervals, treat it as a tie and spend compute elsewhere.