Which Study Do You Trust? Weighing Conflicting Evidence
A student is comparing two passages about school phone bans. Passage A describes one school of 900 students where, after a ban, average marks rose 12% in one term; the figure comes from the principal's newsletter, which also announces the ban as "a resounding success." Passage B describes a district of 12,000 students where researchers found no statistically significant change in marks; it notes that the ban took effect in the same month the district switched to a new bell schedule and a later start time. The student must decide which passage provides stronger evidence, identify the specific weakness in the weaker one, and state what further information would actually change her mind. She has 25 minutes and must write in full sentences.
1. Which passage offers stronger evidence for the claim that phone bans raise achievement? Identify the precise weakness in the other passage rather than simply calling it biased.
Read the worked answer
Passage B is stronger, but note how carefully it must be phrased: it is stronger evidence about the effect of a ban, not proof that bans fail.
Passage A's precise weakness - three of them, in ascending order of seriousness:
- Sample size. 900 students in one school, one term. A 12% swing is well within the range that ordinary cohort variation can produce.
- Source and interest. The newsletter is published by the person whose initiative the ban is. This is not proof of dishonesty; it is proof that the number was selected for public relations rather than designed for testing.
- No comparison group. We see marks after the ban but not what those marks would have done anyway. "Rose 12%" has no baseline in the passage.
Passage B's strength is not its size alone. It is that the passage admits a confound: a later start time is a well-known influence on adolescent achievement, and it changed in the same month. Admitting a rival explanation is a mark of credible reporting.
The trap: the weaker passage is more vivid. Specific numbers and a named school feel concrete. Vividness is not evidence, and the student should say so explicitly.
2. Passage B's confound cuts both ways. Explain how it weakens Passage B's conclusion as well as Passage A's, and state what information would resolve the question.
Read the worked answer
It is tempting to use the bell-schedule detail as a weapon against Passage B - "their null result is contaminated too." That is correct, and it is the point.
The confound does not favour either side. If the district changed start times at the same moment as the ban, then the study cannot separate the two causes. A null result under those conditions is genuinely ambiguous: the ban might have had no effect, or it might have had an effect that the later start time masked or offset. Passage B is more honest, but honesty about a confound is not the same as a clean finding.
What would resolve it: a comparison group that did not change bell times, measured over the same period; or achievement data from the terms before the ban, to establish a trend line. Either would let a reader subtract the schedule change.
The general principle to state in the answer: conflicting sources are not resolved by picking the more likeable one. They are resolved by asking which rival explanations each study has ruled out - and if neither has, the honest verdict is "undetermined."