Item 17, Erased Marks, and a Purged Records Drive
You coordinate a 40-hour fire investigator course. The written exam has 40 items and an 80% cut score. Twenty-six students tested; nine scored between 62% and 78%. The item analysis shows item 17 - on distinguishing the standard for an accidental determination from the standard for an incendiary one - was answered correctly by 7 of 26 (difficulty .27), with a point-biserial of -0.21; the strongest students disproportionately chose distractor C. Item 31 had a difficulty of .95. A proctor's note states that one candidate's answer sheet shows erased and re-marked responses on items 12 through 19, and that candidate's total is 2 points above the cut score. Separately, the registrar emails that skills-check sheets, which policy says to retain three years, lived on a shared drive that IT purged at 90 days; the state licensing body is now auditing a class delivered 20 months ago.
1. Before you decide whether to remediate students or change the item, how do you diagnose item 17? What are your options, and what is the risk of each?
Read the worked answer
A negative point-biserial is a signal about the item, not about the students. High performers choosing one distractor usually means one of three things: the key is wrong, the stem is ambiguous or double-barrelled, or the correct concept was taught incorrectly or not at all.
Diagnose in order. First, check the keyed answer against the source material and the course's own lesson content. Second, read the item cold: does the stem ask one question, is distractor C actually defensible under a reasonable reading, is there a "not" or an absolute? Third, check the instructor's delivery notes.
Options and risks. If the key is wrong, rescore all 26, re-apply the cut score, and document the correction - the risk of not doing it is failing students who answered correctly. If the wording is ambiguous, either revise and void the item for this cohort or accept two defensible answers with documentation; deleting the item to raise scores without cause erodes the exam's link to the job. If the item is sound but was mis-taught, keep the item, remediate, and fix the lesson plan. Deleting a sound item because scores look bad is the one option that is indefensible. Item 31 at .95 should also be reviewed for being too easy to discriminate anything.
2. What do you do about the erased marks on items 12–19 for a candidate who sits 2 points above the cut score?
Read the worked answer
Do not accuse, and do not silently pass. The proctor's note is an observation, not a finding. Preserve the sheet - do not re-scan over it or let it leave the secure area; photograph it and log the chain of custody. Compare the erasure pattern against the seating chart and the proctor's timing log: were those items answered during a period when the proctor was out of the room, and was the candidate seated where another sheet was visible?
Follow the published testing-integrity procedure. That normally means a written statement from the proctor, notice to the candidate, and an opportunity to explain - some erasures are ordinary second-guessing, and many candidates revise answers. Do not alter the recorded score on suspicion alone; a score change without due process exposes the program to a legitimate appeal.
Practical resolution: hold the result pending review, offer a supervised retest on an alternate form, and document the disposition. If the review clears the candidate, the record should say so plainly. If it does not, the program - not the instructor - makes the determination. Your job is to preserve evidence and follow process, not to reach a verdict in the hallway.
3. The licensing body wants records for a class from 20 months ago, and the skills sheets were purged at 90 days. What is your response, and what do you fix?
Read the worked answer
Reconstruct honestly; never recreate. Assemble what genuinely exists: the learning management system's completion records, the instructor gradebook, the sign-in roster, exam answer sheets and item analyses, certificates issued, and any emailed or scanned copies instructors kept. Provide a written statement explaining the retention policy, the actual purge, and exactly which records are unavailable. Fabricating or backdating a skills sheet is a certification-integrity and possibly a legal problem far larger than a missing form.
Tell the auditor before they find it. A disclosed gap with a corrective plan reads very differently from a gap discovered during an audit.
Then fix the system, not just the file: move skills sheets into the controlled records system with the correct retention period, remove automatic deletion, restrict who can purge, and put an annual retention audit on the calendar with a named owner. Confirm in writing to IT what the retention requirement is, since the purge was a default setting rather than a decision. Finally, check whether other cohorts used the same shared drive - the audit has probably just shown you the size of the problem.