Optional Retrieval Practice and a Causal Claim
Researchers followed 1,200 ninth graders in eight schools for one year to test whether a retrieval-practice app improved retention. Teachers were told to use the app 'as they saw fit.' At year's end, students who had used the app at least three times per week scored 0.3 standard deviations higher on a district science test than students who used it less. The researchers concluded that the app improves retention. A critic noted that the app was optional and that students who chose to use it frequently may have been more conscientious. The researchers replied that their model controlled for prior science grades. The critic responded that prior grades do not measure study habits. Neither side disputed the test's reliability. The district is now deciding whether to require the app for all students.
1. What is the researchers' main conclusion, and which evidence is offered for it? Distinguish the conclusion from the method.
Read the worked answer
Conclusion: the app improves retention, which is a causal claim. Evidence: at year's end, students who used the app at least three times per week scored 0.3 standard deviations higher on a district science test than less frequent users. The method was a longitudinal comparison, not an experiment: teachers chose how to use the app, and students' usage differed. The 0.3 SD is the size of the association, not proof of cause. The researchers' control for prior science grades is a method, not the conclusion. A precise restatement would be: 'Among students in these eight schools, more frequent app use was associated with higher science scores, conditional on prior grades.' The leap to 'the app improves retention' requires that frequent use not be a marker of other advantages.
2. What does the critic's reply imply about the sufficiency of controlling for prior grades?
Read the worked answer
The critic's reply implies that controlling for prior grades does not close the self-selection gap. Prior grades measure prior achievement, but conscientiousness, parental support, study time, and motivation can affect both app use and test scores. If those traits are omitted, the 0.3 SD difference may overstate the app's effect. The reply does not prove the app is ineffective; it says the researchers' control is insufficient for the causal claim. It also shifts the burden: the researchers must either measure study habits or use a design that breaks the link between self-selection and use. The critic's position is therefore about internal validity, not about the test's reliability, which neither side disputes.
3. Which additional piece of evidence would most directly address the critic's concern, and what would it show?
Read the worked answer
Random assignment of students or classes to required use versus normal practice, with the same test, would most directly address the concern. If required-use students outperform, the self-selection explanation weakens because use is no longer chosen. If they do not, the original association is likely confounded. A weaker but useful alternative is a direct measure of study habits, such as time logs or teacher ratings, added to the model; that would show whether the app coefficient survives. Random assignment is stronger because it handles unmeasured traits, not just the ones the researchers thought to measure. The district should note that requiring the app changes the question from 'what happens when students choose it' to 'what happens when all students use it,' which is the policy question it faces.