Equating Two Administrations of the Exit Exam
An exit examination is offered in spring and autumn. The spring form had 100 items and a cut score of 62. The autumn form has 100 new items and 20 anchor items carried over from spring. The autumn cohort is larger and appears weaker on prior coursework grades. The examination board wants to keep the same reported cut score of 62 so that 'the standard does not change.' A psychometrician reports that the autumn form is easier than the spring form by about 4 raw-score points on the anchor items.
1. Explain why keeping the reported cut at 62 may be wrong, and what should be done instead.
Read the worked answer
Why 62 may be wrong: The cut score is a point on a particular form's raw-score scale. If the autumn form is easier, a raw score of 62 on the autumn form represents a lower level of ability than 62 on the spring form. Keeping 62 would lower the standard in practice, even though the number looks unchanged.
What to do:
- Equate the forms using the 20 anchor items. Because the autumn form is easier by about 4 points, the equivalent cut on the autumn raw scale is approximately 62 + 4 = 66 (subject to the equating method and its standard error).
- Choose a method: anchor-item equating (e.g., Tucker or mean/sigma, or IRT linking if the model fits). With only 20 anchors, check anchor reliability and item drift; if anchors are unstable, the equating is weak.
- Report on a common scale so that pass/fail means the same thing across sittings.
Key point: 'The standard does not change' means the ability required does not change, not that the raw number stays fixed.
2. The board worries that raising the autumn cut to 66 will fail more candidates from an already weaker cohort. How should this concern be addressed?
Read the worked answer
Address it as a validity issue, not a political one.
- The purpose of the exam is licensure, so the cut must reflect minimum competence, not cohort ability. Adjusting the cut to protect a weaker cohort would defeat the purpose and is exactly the norm-referenced error the board should avoid.
- Verify the cohort claim: Prior grades are a weak predictor; check whether the autumn cohort's lower grades reflect different admissions or different coursework. Do not let an assumption drive the standard.
- Provide evidence: Report the equated cut with its confidence band and show how many candidates fall within the band. Borderline cases can be reviewed individually.
- Support, don't lower: Offer remediation and a re-sit for failing candidates. The correct response to a weaker cohort is support, not a lower bar.
Communication: Explain to the board that a fixed raw number across unequal forms is not fairness; it is inconsistency.
3. What evidence would you require before accepting the 4-point adjustment?
Read the worked answer
Evidence checklist:
- Anchor integrity: The 20 anchor items were not exposed, not revised, and behaved consistently (no drift in difficulty).
- Anchor representativeness: The anchors cover the blueprint domains and difficulty range, not just easy items.
- Sample size: Both cohorts are large enough for stable item statistics; report standard errors of equating.
- Method transparency: The equating method is pre-specified and documented; results are cross-checked with a second method if possible.
- Model fit: If IRT is used, check fit and dimensionality.
- Sensitivity: Report the cut under plausible alternative assumptions (e.g., 3-5 point adjustment) so the board sees the uncertainty.
Decision rule: Accept the adjustment if the anchors are sound and the equating standard error is small relative to the adjustment; otherwise treat the cut as provisional and flag the sitting.