AI Writing Assistant and Mock Results: What the Table Can and Cannot Show
A school with 200 students in the final year of an IB programme surveyed all of them about how often they used an AI writing assistant and compared this with their result in the November mock examinations. Students who used the assistant three or more times per week: 62 passed, 38 did not. Students who used it less than three times per week: 78 passed, 22 did not. The deputy head presents the table to the governors and says the data prove that heavy use of AI tools lowers pass rates by 16 percentage points, and proposes blocking the tool on school networks. A governor asks whether the survey can support that conclusion. The response rate was 100%, but the survey was anonymous and the 'three or more times per week' category was chosen after the results were known.
1. Carry out a chi-squared test of association at the 5% significance level, stating hypotheses and checking conditions.
Read the worked answer
H0: frequency of use of the assistant and mock result are independent. H1: they are associated (dependent).
Row totals are 100 and 100; column totals are 140 passed and 60 did not. Expected count = row total × column total / 200, so every cell is 70 or 30.
(\chi^2 = \dfrac{(62-70)^2}{70} + \dfrac{(38-30)^2}{30} + \dfrac{(78-70)^2}{70} + \dfrac{(22-30)^2}{30})
(= 0.914 + 2.133 + 0.914 + 2.133 = 6.09).
df = (2−1)(2−1) = 1. Critical value at 5% is 3.84. Since 6.09 > 3.84, reject H0: there is evidence of an association.
Conditions: all expected counts are at least 5 (minimum is 30), the sample is the whole year group so there is no non-response problem, and each student appears once. The weakest condition is independence between students: students in the same class share teachers and compare notes, so the effective sample size is somewhat smaller than 200 and the p-value is optimistic. The result should be read as 'unlikely to be chance if the model holds', not as proof of a mechanism.
2. Interpret the table using conditional probabilities, and quantify the uncertainty in the 16-point difference.
Read the worked answer
Conditioning on use:
- (P(\text{pass} \mid \text{heavy use}) = 62/100 = 0.62)
- (P(\text{pass} \mid \text{light use}) = 78/100 = 0.78)
- Difference = 0.16, which is the '16 percentage points' the deputy quotes.
Conditioning the other way, (P(\text{heavy use} \mid \text{pass}) = 62/140 = 0.443), while (P(\text{heavy use} \mid \text{fail}) = 38/60 = 0.633) - the same association seen from the other side, and a reminder that the table cannot say which variable precedes the other.
A rough 95% confidence interval for the difference in proportions:
(SE = \sqrt{\dfrac{0.62(0.38)}{100} + \dfrac{0.78(0.22)}{100}} = \sqrt{0.002356+0.001716} = 0.0638)
(0.16 \pm 1.96(0.0638) = 0.16 \pm 0.125), i.e. about (0.035, 0.285).
So the true gap could plausibly be as small as 3.5 points. The test says 'probably not zero'; it does not say the effect is large, and the interval is wide because each group has only 100 students.
3. Judge the deputy head's causal claim and recommend what evidence would be needed before blocking the tool.
Read the worked answer
The deputy's claim has three separate problems.
First, the design is observational. An association of this kind is fully compatible with confounding: students who are already struggling may reach for a writing assistant more often, which would produce exactly this table with the tool doing no harm at all. Prior attainment, subject choices, teacher and timetable are all plausible confounders and none is controlled.
Second, the category boundary ('three or more times per week') was chosen after seeing the results. Choosing the cut-point post hoc raises the chance of finding a 'significant' difference by chance; with several possible cut-points the effective significance level is larger than 5%.
Third, even a genuine association would not tell the school what happens if the tool is blocked. That is a counterfactual question about an intervention, which the table cannot answer.
What would help: a randomised trial in which volunteers are allocated to tool access or not, or an analysis that controls for prior grades and compares like with like. Until then the defensible statement is that heavy use is associated with a lower pass rate in this sample, with a confidence interval that includes small effects, and no causal claim should be made to governors.