Item Banking After a Compromised Administration
A national exit examination for medical technology graduates was administered at 14 sites to 2,180 candidates. Two weeks later, a proctor at one site reports that a sealed envelope containing 40 of the 120 scored items was opened 90 minutes before the start and photographed by a candidate. Those 40 items were used at all 14 sites. Scores have been released. Pass mark is 60% of total score; the examination is offered twice yearly and is required for licensure. The examination committee must decide how to handle results while preserving defensibility and fairness, with only one backup form of 60 new items available and a two-month window before the next scheduled administration. Legal counsel advises that any remedy must be applied consistently across all candidates.
1. Recommend an immediate course of action for the released scores and explain the psychometric and fairness reasoning behind it.
Read the worked answer
Action: Do not simply void the entire cohort or simply ignore the breach.
- Quantify the exposure. Determine how many of the 40 compromised items were scored (all 40 were) and their weight in the total (40/120 = 33.3%).
- Re-score without the compromised items. Compute each candidate's percentage on the 80 secure items only, and set an equivalent cut score. Because the secure subset is shorter, the cut must be re-derived (e.g., by a modified Angoff or by equating the 80-item scale to the original standard), not copied as a flat 60%.
- Decision rule: Candidates clearly above or below the cut on the secure subset retain their outcome. Candidates whose classification changes (borderline band) are flagged for the next administration.
- Consistency: Apply the same rule to all 2,180 candidates, including those at unaffected sites, because the items were common to all sites.
Reasoning: The breach contaminates only 33% of the score, so discarding all scores is disproportionate and harms candidates who did not benefit. Ignoring the breach rewards exposure and undermines validity. Re-scoring on secure items preserves most measurement information while removing the compromised variance. Documenting the equivalence method protects defensibility.
2. The committee proposes reusing the 60-item backup form for all candidates at the next sitting. Evaluate this proposal.
Read the worked answer
Evaluation: acceptable only with safeguards.
- Coverage: 60 items across the full blueprint may under-sample content domains; check that each domain retains minimum item counts. If not, supplement from the secure 80-item pool.
- Exposure: The backup form has never been administered, so exposure is zero - a strength.
- Equating: A new form cannot be reported on the old scale without a link. Include common items (anchor items) from the secure pool so scores can be equated; otherwise pass/fail decisions across sittings are not comparable.
- Standard setting: Re-derive or confirm the cut score for the new form using a documented standard-setting method rather than assuming 60% transfers.
- Security: Rotate items after use; 60 items is a small bank and will be compromised quickly if reused repeatedly.
Conclusion: The proposal is workable if anchor items, blueprint coverage, and a documented cut score are added; otherwise it creates a new comparability problem.
3. Draft two concrete procedural controls to prevent recurrence, each tied to a specific failure in this scenario.
Read the worked answer
Sealed-packet chain of custody. Failure: envelope opened 90 minutes early and photographed. Control: tamper-evident seals with serial numbers logged at dispatch and verified at receipt; two-person sign-off before opening; opening no earlier than a fixed interval before start; any early opening triggers a documented incident report and holds the affected items from scoring.
Item-level exposure tracking. Failure: the same 40 items were used at all 14 sites with no tracking. Control: an item exposure register recording every administration, site, and date; automatic retirement or rotation after a set number of exposures; and a rule that any item involved in a security incident is quarantined from future forms until reviewed.
Both controls convert a one-time judgment into a repeatable process, which is what defensibility requires.