Defective Item on a High-Stakes Exam
A 150-item multiple-choice certification exam is administered over a two-week window. After the window closes, psychometric review flags Item 87: its point-biserial correlation is 0.02, and 78% of high-ability candidates (top quartile) answered it incorrectly, while 65% of low-ability candidates (bottom quartile) answered it correctly. The item asks about a specific regulatory clause that was updated six months ago, but the item still references the old clause. The exam is used for licensure, and candidates are told they will receive scores within 30 days. The passing standard is set at 70% (105 correct out of 150). The item is worth 1 point. The exam has been administered before with this item, and it performed well then (point-biserial = 0.35).
1. What is the most likely cause of the item's poor performance, and what should be done about it for this administration?
Read the worked answer
The item's negative discrimination (high-ability candidates more likely to get it wrong) and low point-biserial suggest the item is now miskeyed or ambiguous due to the regulatory update. The fact that it performed well previously (point-biserial = 0.35) indicates the content changed, not the item's inherent quality. For this administration, the item should be reviewed by subject matter experts to confirm the key. If the key is wrong or the item is now ambiguous, the item should be removed from scoring (not counted) and the exam rescored out of 149 points. Alternatively, if the item is still defensible but the old clause is no longer correct, it could be double-keyed if two options are defensible, but that is less likely. Removing the item is the safest action to maintain fairness. The passing standard should be adjusted proportionally (70% of 149 = 104.3, so 105 correct may still be required, but the cut score should be recalculated based on the new total). Candidates should be notified of the delay if rescoring takes time, but within the 30-day window, it is feasible.
2. If the item is removed, how should the passing standard be handled, and what communication is needed?
Read the worked answer
The passing standard was set at 70% of 150 = 105. If the item is removed, the total becomes 149. The cut score should be adjusted to maintain the same standard. Simply taking 70% of 149 = 104.3, which rounds to 104 or 105. However, the original standard was based on a criterion-referenced approach, so the cut score should be recalculated using the same method (e.g., modified Angoff) on the remaining 149 items, or proportionally adjusted: 105/150 = 0.70, so 0.70 * 149 = 104.3, meaning 105 correct is still 70.5%, slightly higher. To be fair, the cut score could be set at 104 (69.8%) or 105 (70.5%). The program should document the rationale. Candidates should be informed that one item was removed from scoring and that scores will be based on 149 items. If the delay affects the 30-day promise, communicate proactively. Transparency about the removal (without revealing item content) maintains trust.