Item 94, Nine Days Out
You are the assessment lead for a 180-item professional certification exam scheduled in nine days for 3,100 candidates. A candidate appeal from the previous window argues that item 94 has two defensible answers. You check: the keyed answer matches the reference text's current wording, but the distractor the candidate chose is supported by the newer edition of the same reference - an edition your item writers used for only 40 of the 180 items before switching back mid-cycle. Item 94's point-biserial is 0.04, and 61% of candidates chose the distractor. Fixing the item takes one hour; reprinting the booklet costs the remaining print budget and a four-day delay that would push venue hold fees past the budget line. The vendor can print a one-page erratum insert in 48 hours.
1. What do you do with item 94 for this window? Work through the options before choosing.
Read the worked answer
Option A - keep it as keyed. Fails. A point-biserial of 0.04 says the item does not discriminate; 61% choosing the distractor says the key is inconsistent with how competent candidates read the reference. The key matches one edition, but your writers used both editions, so 'defensible' depends on which book a candidate happened to study.
Option B - accept both answers. Cheap and superficially fair, but it rewards the 61% who read the newer edition and the minority who guessed, and the item still contributes nothing measurable. It fixes the complaint, not the item.
Option C - drop the item and score out of 179. Psychometrically clean, needs no reprint, and can be applied uniformly. Requires a pre-published scoring rule so no candidate can later allege an ad hoc adjustment.
Option D - reprint corrected. Solves the item but buys a four-day delay and consumes the print budget.
Decision: C, plus an erratum insert. Drop item 94 from scoring for all candidates and use the 48-hour insert - or a standard proctor announcement at session start - to tell candidates the item is void and to answer only if they wish. Cost is low, the harm is removed uniformly, and no delay is incurred. Choosing D spends real money to solve a problem C solves for free.
2. Name the process failure this exposes - beyond 'one bad item' - and the two controls you would add.
Read the worked answer
The failure is an uncontrolled reference-edition change mid-cycle. Somewhere, 40 items were written against one edition and 140 against another, with no registry recording which item is keyed to which edition. That is not a proofreading miss; it is a missing control at the point of change.
Control 1 - freeze and register. One governing reference edition per exam form, recorded in an item register alongside each item, with a defined change-control procedure. Any mid-cycle edition change triggers a mandatory re-verification of affected keys, not a judgement call by a writer.
Control 2 - automatic flagging before the window. Any item with a point-biserial below roughly 0.10, or with a distractor attracting more than about 40% of responses, should surface for content review before the next administration. Item 94 would have been caught on statistics alone without any appeal.
A third, weaker but worth noting: the appeal was triaged after the window closed rather than during it. An unresolved dual-key allegation is cheap to investigate in-week and expensive to investigate after 3,100 candidates have sat the form.
3. What do you tell the appellant, and what do you not promise?
Read the worked answer
Tell them: you reviewed the item, agreed that two answers are defensible because the governing reference changed mid-cycle, and that item 94 has been excluded from scoring for every candidate in this window - so their result is not affected by it either way. Acknowledge that the appeal was correct and that the underlying process is being changed. Be specific about the exclusion; vagueness reads as containment.
Do not promise: an individual score change before re-scoring is complete and the scoring rule is published. Do not concede that the key itself was wrong in a way that implies prior windows are invalid - that is a live question, but it belongs to a formal review with the psychometrician, not to an appeal response. Do not discuss other candidates' responses or the 61% figure in a way that identifies anyone. Do not imply the exclusion is a goodwill gesture; it is a correction.
Two things worth doing while you write: flag internally that the same mid-cycle edition switch may have affected earlier windows, so the retrospective review starts now rather than after a second appeal; and keep the response in the appeal file with the dated scoring decision attached, because an exclusion applied uniformly is defensible and one applied quietly is not.