Hypothesis Testing
This domain tests whether a claim about a population is supported by sample evidence. Expect items on setting up hypotheses, choosing the correct test statistic, interpreting p-values, and understanding the trade-off between error rates and power.
The framework
- State the null hypothesis, usually no effect or no difference, and the alternative that answers the research question.
- Fix alpha before looking at the data, compute the test statistic, then either compare it with a critical value or compare the p-value with alpha.
- Two-tailed tests split alpha across both tails. A one-tailed test must be justified by the research question in advance, not chosen after seeing the direction of the result.
Errors and power
- Type I error: rejecting a true null, with probability alpha. Type II error: failing to reject a false null, with probability beta.
- Power is 1 minus beta, the chance of detecting a real effect. It rises with larger effect size, larger n, larger alpha and smaller variability.
- A p-value is the probability of data at least as extreme as observed, assuming the null is true. It is not the probability that the null is true.
Common test statistics
- One-sample t: t = (x-bar minus mu0) divided by (s divided by root n), with n minus 1 degrees of freedom.
- Paired data: reduce to a one-sample t on the differences; do not treat paired observations as independent.
- Two independent samples: difference in means over the standard error, using a pooled or Welch version.
- One-proportion z: z = (p-hat minus p0) divided by the root of p0(1 minus p0)/n, using the hypothesised p0 in the denominator.
Worked example
Test the null that mu = 62 against a two-sided alternative, with n = 16, x-bar = 68, s = 10 and alpha = 0.05. The standard error is 2.5, so t = 6/2.5 = 2.40 with 15 degrees of freedom. The critical value is 2.131, and 2.40 exceeds it, so reject the null; the p-value is about 0.03. A 95% interval, 68 plus or minus 2.131 times 2.5, gives 62.67 to 73.33, which excludes 62 and confirms the decision.
Traps
- Reading a p-value as the probability that the null hypothesis is true.
- Concluding that failing to reject proves no effect.
- Switching from two-tailed to one-tailed after inspecting the data.
- Using p-hat instead of p0 in the standard error of a one-proportion z test.
- Running many tests without adjusting alpha for multiplicity.