The Disappearing Distractors: Item Bank Drift After a Content Update
A certification program for industrial hygienists updated its reference standard in March. In April, the psychometric team reviewed 120 four-option items that had been flagged because their distractor selection rates changed. Before the update, each of these items had a mean distractor endorsement of 18% (range 9-31%). After the update, the mean fell to 6% (range 0-11%), and 41 items now have at least one distractor selected by fewer than 2% of candidates. The exam has 150 items and a passing standard set two years ago. The item writers argue the distractors are still plausible because they reflect common misconceptions; the psychometrician argues the items are now effectively three-option items and should be retired. The program director wants to keep the items because replacing 41 items would exceed the annual item-writing budget.
1. Evaluate the psychometrician's claim that these items are now effectively three-option items. What evidence would you require before agreeing or disagreeing?
Read the worked answer
The claim is plausible but not established by distractor rates alone. A distractor selected by <2% of candidates contributes almost nothing to discrimination, and a four-option item with three live options behaves statistically like a three-option item: guessing floor rises from 25% to 33%, which compresses the lower end of the score scale and can slightly inflate scores for low-ability candidates.
Before agreeing, I would require:
- Item-level discrimination (point-biserial or IRT a-parameter) before and after the update. If discrimination held steady or improved, the dead distractor is cosmetic; if it dropped, the item is degraded.
- Distractor-level discrimination - are the low-rate distractors chosen disproportionately by high or low scorers? A distractor chosen mainly by high scorers is a key error, not noise.
- DIF analysis by relevant subgroups, since a content update can make a distractor systematically unattractive to one group.
- The misconception mapping the writers cite - is there documented evidence that the misconception still exists in the target population?
Only items failing on discrimination plus dead distractors should be retired; the 41-item count is a ceiling, not a floor.
2. The director proposes keeping all 41 items for one more administration and monitoring. What are the risks, and what conditions would make a one-cycle deferral defensible?
Read the worked answer
Risks of deferral:
- Score-scale drift. With 41 of 150 items (27%) behaving as three-option items, the effective guessing floor rises materially. If the cut score is not re-derived, marginal candidates get a small but systematic boost that is an artifact, not competence.
- Reliability erosion. Dead distractors reduce item variance; if the affected items cluster on one content domain, subscore reliability for that domain degrades unevenly.
- Compounding. A one-cycle deferral becomes permanent by default unless a hard trigger is set.
A one-cycle deferral is defensible only with: (a) a written remediation plan with owners and dates; (b) per-item discrimination thresholds agreed in advance (e.g., retire if point-biserial <0.20 or distractor <2% for a second consecutive administration); (c) a sensitivity analysis showing the cut score decision is unchanged when the 41 items are removed; (d) confirmation that no content domain loses coverage below the blueprint minimum if items are pulled; and (e) a budget line for replacement items in the next cycle. Without (c), the deferral is not defensible because the passing decision itself may be affected.