Two colleges and a press release about endorsements
A researcher wants to know whether Year 13 students at Kauri College are more likely to gain a Merit or Excellence endorsement than students at Rimu College. From Ministry records she takes a random sample of 120 Kauri students and 100 Rimu students who completed Year 13 in the previous year; 75 of the Kauri students (62.5%) and 51 of the Rimu students (51.0%) gained Merit or Excellence. A bootstrap 95% confidence interval for the difference in proportions (Kauri − Rimu) is (−1.8%, 24.8%). All students in the sampling frame appear in the records, so there is no non-response. The Kauri principal issues a press release saying 'Our students outperform Rimu students by 11.5 percentage points'. Both schools are co-educational state schools; Kauri draws from a wealthier suburb and offers Cambridge as well as NCEA.
1. Interpret the 95% confidence interval and evaluate the principal's statement.
Read the worked answer
Interpretation: the interval (−1.8%, 24.8%) is a range of plausible values for the true difference in the proportion of students gaining Merit or Excellence at the two colleges. If this sampling procedure were repeated many times, about 95% of the intervals produced would contain the true difference.
Does it support the principal?
- The point estimate is 62.5% − 51.0% = 11.5 percentage points, which is correctly calculated.
- But the interval includes zero, and even includes small negative values. At the 5% level, 'no difference' is a plausible explanation of the data, as is Rimu being slightly ahead. The data are compatible with a wide range of real differences.
- The margin of error is about ±13 percentage points, which is large because the samples are only 100–120 students and the proportions are in the middle of the scale.
Verdict: the press release reports a sample statistic as though it were the population value. A defensible statement is: 'In our samples, Kauri students were 11.5 percentage points more likely to gain Merit or Excellence, but with this sample size the true difference could plausibly be anywhere from about −2 to +25 percentage points.' The claim of superiority is not established by these data.
2. Even if the interval had excluded zero, what else would prevent a causal conclusion about teaching quality?
Read the worked answer
This is an observational comparison, not a randomised experiment. Students were not allocated to schools, so the groups differ in many ways that also affect endorsement rates:
- Intake and decile: Kauri draws from a wealthier suburb; prior achievement and out-of-school support are strongly related to endorsement and are not measured here.
- Course and assessment mix: Kauri offers Cambridge as well as NCEA. Endorsement depends on which standards and courses students attempt; a school with a different course structure can have a different endorsement rate without any difference in teaching.
- Selection and mobility: families who move into the Kauri zone, or who choose the school for its Cambridge pathway, are not a random subset of the population.
- Other confounders: class sizes, staffing stability, subject offerings, and whether students are encouraged to attempt Merit/Excellence standards.
A statistically detectable difference would justify the claim that endorsement rates differ, not that teaching caused the difference. The study also only speaks about these two schools in that one year - a random sample of records from two schools is not a random sample of New Zealand schools, so generalising to 'similar schools' is already a stretch.
3. The principal asks for a 99% confidence interval 'so we can be more sure the difference is real'. Explain what would actually change and what she should do instead.
Read the worked answer
A 99% interval is wider than a 95% interval on the same data - roughly 1.3 times as wide for these sample sizes. So the interval would stretch further below zero, making it less supportive of the principal's claim, not more.
This reflects the trade-off: higher confidence that the interval captures the true value can only be bought with less precision. 'More sure' is not the same as 'more evidence of a difference'.
Misunderstandings to correct:
- A confidence interval is not a probability that the null hypothesis is true, and 99% does not mean '99% chance the difference is real'.
- The decision to reject 'no difference' depends on the interval excluding zero at the chosen level - with these data it does not at 95%, so it certainly will not at 99%.
Better options: if the school genuinely wants to detect a difference of about 10 percentage points with confidence, it needs much larger samples - roughly 400+ per school for a margin of error near ±7 points on a difference. The principal should also be asked what decision the interval is meant to inform; if the question is 'is our endorsement rate changing over time?', a within-school trend comparison is far more informative than a two-school comparison.