Four Sections, Two Teachers, One Confounded Comparison
A grade 8 math department of 96 students across four sections wants to know whether 20 minutes of nightly retrieval practice beats 20 minutes of re-reading notes. Mr. Alvarez teaches periods 1 and 2 (49 students); Ms. Bhatt teaches periods 5 and 6 (47 students), which are the accelerated sections. The plan as run: Alvarez's sections did the nightly low-stakes quizzes, Bhatt's sections re-read their notes, both for five weeks, with a common 25-item post-test in week six. On a 100-point pretest, Alvarez's sections averaged 61% and Bhatt's averaged 74%. Post-test averages were 70% for the retrieval sections and 80% for the re-reading sections. Completion records show 54% of nights logged in the retrieval sections, measured by returned exit tickets, and 81% self-reported in the re-reading sections. Two teachers reading the results reach opposite conclusions. The principal has asked for a one-page summary for parents.
1. Explain how two reasonable people read this as opposite results, and state what the data actually support.
Read the worked answer
The two readings come from two different metrics. Compare post-test means and re-reading wins, 80 to 70. Compare gains and retrieval wins, +9 against +6. The fact that the choice of metric flips the winner is itself a symptom: when a result is that fragile, the comparison is not measuring what it claims to.
Both readings are invalid as causal claims. Condition is completely confounded with teacher, with track, and with time of day. Every retrieval student had Alvarez in the morning; every re-reading student had Bhatt in the afternoon and is in the accelerated sequence. There is no within-teacher contrast anywhere in the design, so teacher effects and track effects cannot be separated from the method. The pretest gap of 13 points also makes gain scores unreliable in different ways at the two starting levels: ceiling pressure on the higher group, more room to move for the lower one.
The completion numbers add a second confound - 54% logged against 81% self-reported are not the same measurement, and self-report on a five-week habit is exactly the measure most likely to be inflated. What the data support is a description of what happened in four specific rooms, not a recommendation.
2. Redesign the study using the same four sections, two teachers, five weeks, and one post-test.
Read the worked answer
Randomize within teacher. Alvarez teaches one section retrieval and one re-reading; Bhatt does the same, with the assignment of section to condition drawn at random rather than by period. That single change removes the teacher confound and makes the accelerated track appear in both conditions, so track is balanced rather than fatal.
Use the common 25-item post-test, score it blind to condition, and use the pretest as a covariate rather than relying on raw gain scores, since the groups start at different levels by design.
Fix the fidelity measurement. Use the same instrument for both conditions - returned exit tickets - instead of pairing a behavioral log against self-report. A $0 fix that decides whether the groups actually received different doses.
Script the implementation. If one teacher is more enthusiastic about retrieval practice, that enthusiasm rides along with the condition; a short written protocol for both nightly tasks reduces the differential.
State the limits honestly in advance: the unit of assignment is the class, not the student, so n = 4 and no student-level significance test is appropriate. Frame it as a pilot whose job is to decide whether a larger, multi-school trial is worth running.
3. The department has already run the flawed version, and the principal wants the parent summary to say retrieval practice works. What do you write and what do you do with the data?
Read the worked answer
Do not write that. The parent summary should present both metrics, name the confound plainly - different teachers, different tracks, different times of day, no within-teacher comparison - and describe the exercise as a pilot that produced a planning decision, not a finding about children. Then describe the redesigned version and the term it will run.
That is not an empty compromise. There is a defensible position available: recommend retrieval practice on the strength of the general evidence base, while stating clearly that this particular comparison did not test it. Those are two separate claims, and only one of them is being asked to carry weight it cannot.
Use the existing data for what it can do. The 54% completion figure is a real and actionable finding about implementation - whatever the method, most students were not doing it nightly. That belongs in the summary. Also flag that the 81% figure is self-report and should not be quoted as a completion rate at all.
Finally, protect the record. Keep the raw sheets and the assignment of sections in a file, because the strongest reason to redesign rather than re-argue is a clear account of what the first run could and could not show.