The Study Skills Workshop
A college learning center runs a workshop claiming it improves exam performance. To evaluate it, staff recruit 60 volunteers. All volunteers take a 40-item practice test in September. Those who score in the bottom half (n = 30) are invited to attend the free workshop; those in the top half are not invited. In December, everyone retakes a parallel 40-item test. The workshop group's mean rises from 18 to 27 correct; the comparison group's mean rises from 26 to 28. The center director concludes the workshop caused a 9-point gain and wants to expand it. A skeptical faculty member notes that the workshop group was selected precisely because they scored low, and that students who choose to attend free sessions may differ from those who do not.
1. Identify the two distinct methodological problems in this evaluation and explain how each could produce the observed pattern of results without any real effect of the workshop.
Read the worked answer
Problem 1: Regression to the mean (selection on the dependent variable). The workshop group was chosen because their September scores were unusually low. Extreme scores on any imperfectly reliable test tend to drift toward the average on retest even with no treatment. So part of the 18→27 gain is expected purely from selecting the bottom half. The comparison group, selected for high scores, has less room to rise and drifts toward the mean in the other direction, which flattens their gain (26→28).
Problem 2: Self-selection / non-equivalent groups. Invitation was based on score, so the groups differ at baseline in ability and motivation, and attendance was voluntary. Students who show up to a free workshop may be more motivated or better organized, so motivation is confounded with the treatment. There is no random assignment, so the design is a quasi-experiment, not a true experiment.
Neither problem alone proves the workshop is useless; together they mean the 9-point figure cannot be attributed to the workshop.
2. Design a stronger test. Specify the assignment procedure, what should be measured, and one ethical constraint the center must respect.
Read the worked answer
Design: Random assignment. Recruit students who are eligible (e.g., all volunteers below a cutoff, or all volunteers regardless of score). Randomly assign half to the workshop and half to a waitlist control. Both groups take the same September and December tests under the same conditions.
Measure: The December score (or the September-to-December change) as the dependent variable, compared between groups. Include a baseline covariate (September score) to increase power and to control for regression to the mean statistically. Ideally, blind the scorers to condition.
Ethical constraint: Do not permanently withhold a potentially beneficial service. The waitlist control should receive the workshop after the study ends (a delayed-treatment design), and students must give informed consent and be free to withdraw. Random assignment to a waitlist is acceptable because no one is denied the workshop permanently.
3. The director says, 'Even if it's not the workshop, the students improved, so who cares?' Explain, in terms of internal and external validity, why a psychologist would still care.
Read the worked answer
Internal validity: If the gain is due to regression to the mean or self-selection, the center learns nothing about the workshop's causal effect. It may spend money expanding an inert program while attributing improvement to it. Internal validity asks whether the treatment, not a confound, produced the change.
External validity: Even a real gain in this self-selected, low-scoring volunteer sample may not generalize to the broader student population or to other test formats. The claim 'the workshop works' would be overgeneralized.
The practical point: improvement is not evidence of causation. Without a comparison that rules out the confounds, the director cannot justify the expansion on these data.