Tea and Marks: A Survey That Proves Too Much
A survey during health week at a school in Thrissur collects data from 120 Class XII students on sleep, diet and examination marks. The summary shows that the 40 students who report drinking tea before an examination average 71 marks, while the 80 who do not average 63. A student presents this at morning assembly and concludes: 'Tea improves exam performance; therefore every student should drink tea before the board examination.' The teacher checks the questionnaires. Thirty of the 40 tea drinkers also report sleeping more than seven hours, against only 20 of the 80 non-tea drinkers, and the tea drinkers are mostly from the science batch, whose papers are marked by a different scheme. The data are self-reported, collected once, from a single school. The student insists that a sample of 120 is large, so 'the numbers cannot lie' and the conclusion must stand.
1. Identify the logical error in the assembly conclusion, name it, and explain why the large sample size does not rescue the inference.
Read the worked answer
The error. The student moves from a statistical association to a causal claim, and then to a prescription. In Latin tags, this is cum hoc ergo propter hoc ('with this, therefore because of this'); if the student also thinks the tea came first and the marks followed, it is post hoc ergo propter hoc. Both are varieties of the fallacy of false cause. Even if tea-drinking and marks are genuinely correlated, the conclusion 'tea improves performance' does not follow, and the further leap to 'everyone should drink tea' compounds the error by generalising from one school to all candidates.
Why the confounders matter. The data contain at least two variables that travel with tea-drinking: sleep (30 of 40 tea drinkers sleep over seven hours, versus 20 of 80 others) and batch (science students, marked differently). Either could produce the eight-mark gap on its own. Tea may be a marker of a well-rested, science-batch student rather than a cause of marks.
Why sample size is irrelevant here. Increasing n reduces random error - the sampling wobble. It does nothing about systematic error: confounding, biased selection, self-report inaccuracy, and a single-school sample. A very large biased sample gives a very precise wrong answer. Precision and validity are different virtues, and the student has confused them. The appeal to numbers is an appeal to a feature that does not address the actual defect.
2. Using Mill's method of difference, redesign the study so that it could support a causal conclusion. You have only the 120 students and one examination cycle. State what is held constant, what is varied, and what would count as a confirming or disconfirming result.
Read the worked answer
The logic of the method of difference. If two situations agree in every relevant circumstance except one, and the effect appears in one and not the other, that circumstance is causally implicated.
Design.
- Restrict the population. Take students from a single batch, sitting the same paper, marked by the same scheme. This removes the marking confounder.
- Match on sleep and prior attainment. Pair students with similar reported sleep (say, 7–8 hours) and similar marks in the previous examination. Randomly assign one member of each pair to tea and the other to a placebo drink that looks and tastes similar but contains no caffeine. Matching handles known confounders; randomisation spreads the unknown ones.
- Hold constant the testing conditions - same time of day, same duration, same invigilation, same instructions, and no other caffeine on the day (record and exclude other sources).
- Blind where possible. Students should not know the hypothesis; the examiner marking should not know who received which drink.
- Pre-specify the outcome - total marks - before collecting data.
What counts. If the tea group outperforms its matched placebo partners by a consistent margin across pairs, the method of difference supports a causal role for tea (or its caffeine). If the differences are mixed, or vanish once sleep and prior marks are matched, the original claim is disconfirmed and the association is explained by the confounders.
Caveat. Even a positive result would license only a limited causal claim, for this population and this dose, not the universal prescription 'everyone should'. And a single trial is weak evidence; replication is needed.
3. A second student responds: 'Induction can never prove anything, so the whole survey is worthless.' Evaluate this sceptical reply. Is it a good objection to the assembly conclusion?
Read the worked answer
The grain of truth. Hume showed that inductive inferences presuppose the uniformity of nature, and that this presupposition cannot be demonstrated without circularity. No finite set of observations entails a universal law. So the sceptic is right that the survey cannot prove a causal law in the way a deductive argument proves its conclusion.
Why the conclusion does not follow. 'Not proven' is not 'worthless'. Three points tell against the second student.
First, the objection proves too much. It would destroy not only the tea survey but all empirical medicine, agriculture and engineering, and the student's own reliance on the school timetable. A standard that no empirical inquiry can meet is not a useful standard for evaluating this inquiry.
Second, the survey has a specific, curable defect that has nothing to do with the general problem of induction. It is confounded. The remedy is a controlled design, not a retreat to scepticism. The second student's reply obscures the real error by relocating it to an unanswerable philosophical plane.
Third, a fallibilist or Popperian position gives the right emphasis: science does not prove, it conjectures and refutes. The redesigned experiment can refute the causal hypothesis if matched pairs show no difference. That is genuine cognitive progress without proof. A hypothesis that survives serious attempts at refutation is rationally preferable to one that has not been tested.
Verdict. The sceptical premise is correct but the inference from it is too strong. The right response to the assembly claim is not 'no survey can prove anything' but 'this survey cannot show causation, and here is a better one'.