The Trial of the New Study Method
A researcher tests whether a new retrieval-practice method improves exam scores. She recruits 60 Year 12 students from one school. Students choose whether to join the 'new method' group or the 'usual revision' group. The new-method group meets in the library after school with a tutor; the usual group revises at home. After six weeks, the new-method group scores 8 marks higher on average on a test written by the researcher. The researcher concludes the method works and recommends it for all schools.
1. Identify the threats to internal validity in this design and explain how each could produce the observed difference.
Read the worked answer
Several threats operate at once.
Selection bias: students chose their group, so the groups may differ before the study begins. Students who opt into an extra after-school session with a tutor may be more motivated or already higher-achieving. The 8-mark gap could pre-exist the method.
Confounding: the new-method group gets a tutor and a structured library setting; the usual group revises alone at home. Tutor presence, structure and location are all confounded with the method itself, so we cannot attribute the gain to retrieval practice.
Experimenter bias: the test was written by the researcher, who knows the hypothesis and may have written items closer to the new-method materials.
Maturation and history are less severe over six weeks but not eliminated.
Because these threats all push in the same direction - favouring the new-method group - the 8-mark difference is not good evidence for the method.
2. Describe the key design changes needed to make this a true experiment, and explain what the researcher could and could not conclude after those changes.
Read the worked answer
Key changes:
- Random allocation of students to conditions, so pre-existing differences are distributed by chance rather than by choice.
- Equate the conditions except for the independent variable: both groups should have the same setting, the same tutor time, and the same schedule, with only the revision method differing. Alternatively, give the usual group an equivalent tutor session using their method.
- Use a test written and marked by someone blind to condition, or use a standardised test, to remove experimenter bias.
- Pre-test both groups so gains can be measured rather than just post-test scores compared.
After these changes, the researcher could conclude that, in this sample, the method caused a difference in test performance. She could not conclude that it will work in all schools: one school, one sample and one test limit generalisation. Replication across schools and settings would be needed before a universal recommendation.