Interpreting Results
What this domain tests
This domain tests whether you can read statistical and model output correctly, separate association from causation, quantify uncertainty, and translate findings into defensible business implications. The exam rewards cautious, context-aware interpretation.
Core rules
- Distinguish statistical from practical significance. A small p-value does not imply a large or valuable effect. Report effect sizes and confidence intervals.
- Interpret p-values correctly. A p-value is the probability of data at least as extreme under the null hypothesis; it is not the probability that the null is true.
- Respect uncertainty. Confidence intervals show a range of plausible values. Wide intervals mean the estimate is imprecise, even if significant.
- Do not infer causation from observational association. Consider confounding, reverse causation, selection bias, and Simpson’s paradox.
- Check calibration and residuals. A model can rank well but produce poorly calibrated probabilities; regression assumptions and residual patterns matter.
- Account for multiple comparisons and segment search. Testing many subgroups inflates false positives.
- Distinguish feature importance from causal effect. Importance describes model behavior, not what happens if you intervene.
Worked example
An A/B test reports conversion of 2.3% versus 2.0%, p = 0.04, with a 95% confidence interval for the difference of 0.02 to 0.58 percentage points. The result is statistically significant at the 5% level, but the lower bound is near zero and the interval is wide. If the business requires at least a 0.5-point lift to justify rollout cost, the evidence is inconclusive because most of the interval lies below the threshold. Recommend extending the test or reporting the result as promising but not yet sufficient. Do not claim a guaranteed 0.3-point gain, and do not describe p = 0.04 as a 96% chance the effect is real.
Traps that cost marks
- Saying a p-value is the probability the null hypothesis is true.
- Treating a statistically significant result as automatically business-relevant.
- Claiming causation from a regression coefficient or feature importance.
- Ignoring confidence intervals and reporting only point estimates.
- Cherry-picking a favorable subgroup after many comparisons.
- Forgetting that predictive accuracy does not establish calibration or causal impact.