The Recruit Who Passed Everything Except the Ladder
You are the lead evaluator for a 14-person recruit class. The final practical exam has four stations, each scored by a different instructor using department checklists. One recruit, Candidate R, averages 86 across the class but scores 68 on the ground ladder deployment station. The passing standard is 75 per station. The checklist item is uses proper technique to raise the ladder, with no further description. Evaluator A, who scored R, is known for strictness; Evaluator B is known for giving second chances. Candidate R has an offer contingent on certification, and the academy coordinator says, A fail here will cost us a slot. The department's remediation policy allows one retest after remedial training but does not define the retest conditions. R's file shows a prior shoulder injury disclosed at hiring. You must decide whether to fail R, remediate, or pass, and recommend changes.
1. Analyze the assessment's validity and reliability and decide the immediate outcome for Candidate R.
Read the worked answer
Reasoning approach
The score alone is not enough to fail a candidate on a safety-critical skill, but it is enough to stop and investigate. The instrument has a validity problem: uses proper technique is not an observable standard, so Evaluator A and Evaluator B may be scoring different behaviors. The process has a reliability problem: four evaluators, no calibration, and known differences in leniency. The 68 may reflect a real performance deficit, an evaluator severity effect, or an ambiguous checklist item. All three are plausible.
Immediate steps: review the completed checklist, interview Evaluator A about what was observed, and have a calibrated evaluator observe R perform the same ladder deployment under the same conditions. Check whether R's shoulder injury affects the required technique and whether any accommodation was documented. Do not pass R merely to save the slot, and do not fail R merely because the score is below 75 if the instrument cannot support that conclusion.
If the second observation confirms a critical performance failure, remediation is appropriate under the policy. Define the remediation precisely: practice sessions, the specific steps to correct, and the conditions of the retest. If the second observation shows acceptable performance, the original score is not defensible and R should not be failed on that station. In either case, document the review and the basis for the decision.
2. What changes would you make to the instrument and evaluator process before the next class?
Read the worked answer
Reasoning approach
Rewrite the ladder checklist into behaviorally anchored steps with critical pass/fail items. For example: selects correct ladder; checks for overhead hazards; raises with proper body mechanics; heel and tie-off completed; extension locked; climbing angle within standard; communicates commands. Remove vague words or define them in an evaluator guide.
Calibrate evaluators before the exam. Have all four score the same recorded or live performance and compare results. Discuss disagreements until the group applies the same standard. Use two evaluators for high-stakes stations or audit a sample of scored stations. Rotate evaluators so one strict or lenient evaluator does not determine a candidate's outcome.
Standardize remediation and retest conditions: who provides remediation, how many hours, what must be demonstrated, who evaluates the retest, and what happens if the candidate fails again. Include an appeal route and a process for candidates to review their results.
Collect data on station difficulty and evaluator patterns. If one station has a much lower pass rate, examine the instrument and instruction before assuming candidates are weak. Keep assessment records that show the standard was applied consistently. This protects the department legally and gives candidates a fair process.