The Newsletter Pass-Rate Claim: Which Comparison Survives Scrutiny?
Chiba Minami High School's science club ran two optional revision courses before the second-year mock examination, and the club newsletter, going to print tomorrow, states: 'Course A is clearly the more effective course, because 75% of its students passed, against 67% of those on Course B.'
The raw numbers: Course A enrolled 60 students and 45 passed; Course B enrolled 120 students and 80 passed. The club president, however, checked who had signed up. Course A was chosen by 48 students who had scored above 70% on the previous test (40 of them passed) and by 12 students below that line (5 passed). Course B was chosen by 30 students above 70% (27 passed) and by 90 below it (53 passed). Both courses were taught by the same two teachers over the same number of hours, and students chose freely; nobody was assigned.
The editor wants to keep the item to three sentences and needs a claim that the data can actually support.
1. Calculate the overall pass rate of each course and the pass rate inside each prior-attainment group. Explain what happens to the comparison when the groups are pooled.
Read the worked answer
Overall rates: Course A = 45/60 = 0.750 (75.0%); Course B = 80/120 = 0.667 (66.7%).
Subgroup rates: above 70% previously, A = 40/48 = 83.3%, B = 27/30 = 90.0%; below 70% previously, A = 5/12 = 41.7%, B = 53/90 = 58.9%.
So Course B is ahead in both subgroups and behind overall. The cause is the mix of intakes: 48/60 = 80% of Course A's students came from the stronger group, but only 30/120 = 25% of Course B's did. Strong students pass at a high rate whichever course they attend, so a course that recruited mostly strong students inherits a high headline rate. Pooling weights each subgroup by enrolment rather than by a common standard, so the 75% versus 67% comparison measures intake at least as much as teaching. This is the classic reversal in which an association at the group level points the opposite way to the association inside every subgroup.
2. The editor asks: 'Can we say Course A causes better results?' Give a defensible answer and name the single design change that would most strengthen a causal claim.
Read the worked answer
No, and the data actively undercut the claim. Three obstacles. First, prior attainment is a confounder, and once it is controlled the comparison reverses rather than merely shrinking. Second, allocation was self-selected: students who chose Course A may differ in motivation, timetable or friendship group, none of which was measured. Third, the weaker subgroup of Course A contains only 12 students, so 5/12 versus 53/90 is compatible with a wide range of true differences.
The single most useful change is random allocation of volunteers to A or B, keeping teachers, hours and materials identical, with a common assessment marked blind to course. Randomisation balances the unmeasured variables that self-selection lets in. If randomisation is impossible, the honest fallback is to report subgroup rates, state the confounder explicitly, and avoid any causal verb.
3. Write the newsletter's three-sentence item using the data honestly.
Read the worked answer
A defensible version:
'Course A and Course B gave similar results once students' earlier scores are taken into account: among students who had previously scored above 70%, 83% of Course A and 90% of Course B passed the mock examination. Because students chose their course themselves, these figures describe who enrolled as much as what the courses did. The club will report future results by prior attainment and will consider random allocation.'
Checks a reader should apply: no causal verb such as 'causes', 'proves' or 'is more effective'; the confounder is named in the item itself, not only in the president's notes; no superlative; and the numbers quoted match the computed subgroup rates rather than the flattering pooled figure.