Using Reading scores to place writers
A secondary school proposes to use STEP Reading section scores to place incoming students into 'extended writing' versus 'standard writing' groups, because reading is cheaper to administer than the writing paper. Last year's data: 96 students took both. The correlation between Reading and Writing total scores is r = .42. Of the 22 students placed in extended writing on the strength of a Reading score of 75+ out of 100, 9 were rated below the writing-group threshold by their teacher in week 3, and 4 asked to move down. Meanwhile, 6 students with Reading scores of 60–74 produced strong writing portfolios. The head of department says the correlation is 'good enough' and asks you to sign off the placement rule.
1. Evaluate the proposed use of the Reading score. Give the head of department the numbers that matter.
Read the worked answer
Translate r into what placement actually experiences
- r = .42 means r² ≈ .18: Reading accounts for about 18% of the variance in Writing. Roughly four-fifths of the variation is elsewhere. For a group average that might be tolerable; for classifying individuals it is not.
Give decision accuracy, not correlation
- Of 22 placed in extended writing, 9 (41%) were judged below threshold within three weeks. A rule that misfires on two in five of the students it selects is not 'good enough'.
- The 6 students at 60–74 with strong portfolios are the other error type: false negatives. The head has been shown only half the table.
Name the construct problem Reading and writing share academic language but not the composing processes the writing group is organised around - planning, drafting, revising under a reader's expectations. A reading score cannot see those.
Verdict: do not sign off. Use Reading as a screening flag at most, never as the sole classification instrument.
2. What evidence would you require before signing off, and what is the smallest defensible alternative?
Read the worked answer
Evidence I would require
- A decision-accuracy table, not a correlation: for each possible cut score, how many true positives, false positives, true negatives, false negatives. Then a judgement about which error the school can live with.
- Cross-validated on a second cohort. A rule fitted on the 96 students it was derived from will flatter itself.
- Evidence that the Writing paper itself predicts the outcome the groups exist for (week-3 teacher rating, portfolio quality). If the Writing paper doesn't predict it either, the problem is the grouping criterion, not the proxy.
Smallest defensible alternative
- Keep the Reading score as a flag: 75+ triggers a 20-minute timed writing sample, scored against the group threshold. That costs one short session, not a full paper, and it converts a weak predictor into a valid decision rule.
- For students at 60–74 with strong portfolios, add a teacher-recommendation route so the false-negative group has a path in.
3. The head insists on proceeding. How do you respond, and how do you handle the 9 misplacements?
Read the worked answer
Respond in writing, not in conversation
- Put the accuracy table in a one-page memo: 22 placed, 9 below threshold, 6 strong writers excluded. State clearly that you are not signing off, and state what would change your mind (the evidence list).
- Offer the flagged-sample alternative as a live option with its cost. Refusal without an alternative reads as obstruction; refusal with a cheaper valid option is hard to argue with.
Handle the 9
- Move them without treating the move as failure: 'the placement rule, not you, got this wrong' - and say it to the students, because they will otherwise read it as demotion.
- Do not quietly leave them in place for the sake of the timetable. A group whose entry criterion is known to be invalid will be taught to a level half the class does not need.
- Track what happens to the 6 excluded students. If they are not offered the extension route, the school has documented a second, avoidable error.
Follow up at the end of term with the actual outcome data, so the argument is settled by evidence rather than by seniority.