Hypothesis Testing
This domain tests how to frame a question about a parameter as a choice between two models, and how to interpret the answer honestly.
Core rules
- H0 versus H1; a test statistic; a critical region. The significance level alpha is P(reject H0 given H0 true) - the Type I error rate. A Type II error, probability beta, is failing to reject a false H0; power is 1 - beta.
- The p-value is the probability of a result at least as extreme as that observed, assuming H0 is true. It is not the probability that H0 is true.
- One-sided tests put all of alpha in one tail; two-sided tests split alpha/2 into each tail. Choose before seeing the data.
- Standard tests: likelihood ratio, Wald (estimate minus hypothesised value, divided by its standard error) and score tests. All three are asymptotically chi-square.
- Chi-square goodness-of-fit uses sum of (O - E)^2 / E, with degrees of freedom equal to the number of groups minus the number of parameters estimated from the data minus one.
- A confidence interval contains exactly those parameter values not rejected by a two-sided test at the corresponding level.
Worked example
Last year q = 0.01 and we suspect it has risen. With 1,000 lives and 15 deaths, H0 gives mean 10 and standard deviation sqrt(1000 x 0.01 x 0.99) = 3.147. So z = (15 - 10)/3.147 = 1.59, giving a one-sided p-value of 0.056. With a continuity correction, z = (14.5 - 10)/3.147 = 1.43 and p = 0.076. An exact binomial calculation gives about 0.084. All three agree: no rejection at 5%, but the evidence is borderline, not absent.
Traps that cost marks
- Reading a p-value as the probability that the null hypothesis is true.
- Swapping the one-sided and two-sided critical values.
- Treating a non-significant result from a small sample as proof of no effect - check the power.
- Miscounting degrees of freedom after estimating parameters.