Three Evaluators, Three Different Passing Lines

The results are not reliable enough to release as pass/fail decisions. The same performance standard produced pass rates of 94%, 61%, and 78% across three evaluators. That spread is a large evaluator effect, not evidence that the candidates differed that much. Evaluator A appears lenient: giving 4s to candidates who missed safety checks suggests the rubric is not being applied as intended, especially if safety is a critical behavior. Evaluator B appears severe and may have committed a procedural error by stopping a candidate at 12 minutes without confirming the time rule and without narrative support for a 1. Evaluator C’s clustered 3s with no comments provide no defensible evidence for either pass or fail.

Validity is also weak because the rubric defines “meets” numerically but not behaviorally. Evaluators cannot consistently observe “instructional delivery” if the rubric does not say what a 2 looks like versus a 4. Releasing results tomorrow would expose the center to successful appeals and would not tell the candidates what they need to improve.

The decision: do not release final results. Hold them as preliminary, re-calibrate the evaluators, and re-score the video-recorded presentations using a behaviorally anchored rubric. If re-scoring changes a result, follow the appeal process.

First, revise the rubric so each category has observable anchors. For example, “safety checks” might be scored as: 1-omits two or more required checks; 2-omits one; 3-completes all checks but with prompting; 4-completes all checks independently and explains them. Do this for all five categories. Define critical behaviors that cannot be offset by high scores elsewhere, such as unsafe live-fire handling.

Second, run a calibration session with the three evaluators using three recorded presentations that represent low, middle, and high performance. Each evaluator scores independently, then the group compares scores and discusses discrepancies against the anchors. The goal is not forced agreement; it is a common interpretation of evidence. Document the calibration and the final anchors.

Third, re-score all 18 videos independently, preferably with evaluators not assigned to their original candidates. Use two evaluators per candidate or a sample audit if resources are tight. Require a written comment for any score of 1 or 4 and for any critical-behavior failure. Recalculate pass/fail using the 2.5 standard and the critical-behavior rule.

Fourth, compare original and re-scored results. If a candidate’s outcome changes, that candidate receives the re-scored result and an explanation. The original scores remain in the file as part of the audit trail.

Notify the two appellants promptly that the results are under review, not final. Explain the concern: evaluator consistency and rubric clarity, not candidate misconduct. Offer each appellant a copy of the rubric anchors and, where possible, a review of their video with a different evaluator present. Do not promise a pass; promise a defensible process.

For each appeal, create a file containing the original score sheet, the evaluator’s written comments or a note that none were recorded, the video, the re-scored sheet, the calibration record, and the final decision with rationale. If the appeal is upheld, record the new outcome and any required remediation. If it is denied, record the specific evidence that supports the denial against the anchors.

At the program level, create a summary record: evaluator pass rates before and after calibration, inter-rater agreement, changes to the rubric, and the date the revised rubric takes effect. Also record the corrective action for Evaluator B’s time-stop procedure and for Evaluator A’s undocumented 4s. The records should show that the center treated the appeals as a quality-assurance issue. Release final results only after re-scoring and appeal review are complete, and retain all versions so an external reviewer can reconstruct the decision.