The Critical Item and the Item Everyone Missed
You are one of three evaluators at a practical certification exam. Twenty-two candidates rotate through four stations, and you are assigned the SCBA station. Candidate R is a firefighter on your own shift. The rubric lists "demonstrates emergency bypass/purge" as a critical item, and any critical miss fails the station, with remediation and retest permitted after 24 hours. R skips the step, then argues he "did it in his head" and that the step is a technicality. Separately, you run item analysis on the 40-question written exam: 19 of 22 candidates missed item 12, which asked about a hose appliance that appears nowhere in the lesson plan or the practical stations.
1. How do you score R, and how do you handle the fact that he is on your shift?
Read the worked answer
The rubric decides before the candidate does. The item is critical because the failure mode is injury or death - emergency bypass exists for a reason - and what gets scored is the performance, not the intent. I say that once, plainly, without debate: the step was not performed, the rubric fails the station on that item, here is the remediation path and the retest date.
What I do not do is soften the score because he is on my shift, and I do not harden it to prove I am impartial. Both are the same error. I disclose the relationship to the lead evaluator before scoring continues, and where policy permits I ask that a different evaluator score the retest - not because I would be unfair, but because the record has to look as clean as it is. The retest is a full re-demonstration of the station, not a make-up of the missed step, and the failure plus remediation go into the record.
2. Nineteen of 22 missed item 12, which covers material never taught. What is your process, and what will you not do?
Read the worked answer
A miss rate that high is a signal about the item, not the class. First I look at the distractor pattern - if the strongest candidates also missed it, the item is probably testing something outside the curriculum or is ambiguously worded. Then I check the lesson plan. If the appliance genuinely is not in the curriculum, the item is invalid for this exam and goes to the exam committee for review; I do not quietly delete one item on my own authority.
Practically: if the item is invalid, every candidate receives credit and scores are recalculated before anything is released. If the item is valid but the content was badly taught, the content is retaught and the item stays, because lowering the bar would hide a curriculum gap. Either path is documented through the test-review process.
The one thing I will not do is adjust an individual candidate's score to get him over the passing line. That converts a defensible test problem into an indefensible integrity problem.