Evaluating the StudyBoost Press Release
A teacher finds a press release that reads: 'Students who use the StudyBoost app score 27% higher on tests.' The release describes a study in which 38 high school students who had downloaded StudyBoost on their own were compared with 38 classmates who had not. All 76 took a 10-question quiz at home, on their own devices, and reported their scores through a web form; the app users averaged 7.0 correct and the non-users averaged 5.5 correct, a 27% difference. The study was funded by the app's developer, was not pre-registered, and reports a 95% confidence interval for the difference of 0.4 to 2.6 points. The teacher wants students to decide whether the school should buy the app and to practice evaluating evidence the way they will need to as citizens.
1. Identify the main design weaknesses in the study and explain how each could inflate the apparent effect. What would a stronger design look like?
Read the worked answer
Weaknesses and how each inflates the effect:
- No random assignment. Students chose the app themselves, so app users may already be more motivated, more comfortable with technology, or higher-achieving. Selection differences, not the app, can produce a 1.5-point gap.
- No active control. Non-users did not do a matched alternative task, so the comparison is 'app versus nothing,' which mostly measures extra time on task.
- Unblinded, self-reported outcome. Students knew their group and reported their own scores from a home device; they could look up answers or misreport, and expectancy can operate.
- Small sample, single quiz. 38 per group is modest, and one 10-question quiz is a noisy measure.
- Funder and no pre-registration. The developer funded the study and the analysis plan was not fixed in advance, raising the risk of selective reporting.
A stronger design: randomly assign students to StudyBoost or an active control (the same content on paper or a neutral study-skills app), proctor an objective outcome, pre-register the analysis, blind the scoring, measure baseline achievement as a covariate, and use independent funding. That design could support a causal claim; this one cannot.
2. Explain the difference between the relative claim ('27% higher') and the absolute difference, and interpret the confidence interval. Does the result support buying the app?
Read the worked answer
The 27% is a relative difference: 1.5 points divided by the non-user mean of 5.5 points. The absolute difference is 1.5 points on a 10-question quiz, or 15 percentage points, which is the more useful way to state it. Relative framing makes small absolute effects sound large.
The 95% confidence interval for the difference is 0.4 to 2.6 points. The interval excludes zero, suggesting a real difference in this sample, but it is wide: the true effect could be as small as 0.4 points (about 4% relative) or as large as 2.6. A 10-item quiz also has substantial measurement error, so a 1.5-point difference is roughly one or two questions.
For a purchase decision, statistical significance is not enough. The school should ask about cost, whether the effect persists, whether it holds for its own students, and whether an active-control study shows an advantage over ordinary studying. On this evidence, the defensible statement is 'app users scored somewhat higher in an uncontrolled comparison,' not 'the app raises scores by 27%.'
3. Name a plausible confounding variable and describe how it could produce the result without any effect of the app.
Read the worked answer
A plausible confounder is prior academic motivation or achievement. Students who seek out and keep using a study app may already study more, have better organizational habits, or come from homes with more academic support. Those same traits raise quiz scores with or without the app, so the app becomes a marker of motivation rather than a cause of the higher score.
A second candidate is device and home environment: app users needed a device and internet access to download the app, and they took the quiz at home on that device, where they could use notes or search. Non-users may have had less access. Either confounder would produce a gap with no causal effect of StudyBoost.
The fix is design, not statistics: randomly assign students within the same schools and grade levels, measure prior achievement, and use a proctored outcome. Randomization balances measured and unmeasured confounders, which is exactly what an observational comparison cannot do.