The Rubric That Everyone Passes

It is a system problem expressed through rater behavior. The 3.7-point gap is more plausibly a rater effect than a true section difference. The rubric has no behavioral anchors, so 'demonstrates empathy' invites each instructor to apply a private standard; the sample review shows exactly that. Section A counts any reflective statement, while Section B requires reflection plus a follow-up. That is systematic severity and leniency plus construct-irrelevant variance, and it threatens both validity and fairness. The absence of rater training and double-scoring means there is no mechanism to detect or correct it. The chair should not treat the appeal as a public-relations problem or assume all role-play assessment is hopelessly subjective. Performance assessment can be reliable when the construct is defined behaviorally and raters are calibrated. The immediate task is to determine whether the discrepancy is systematic: sample recordings across both sections, have two trained raters score blind, and compare. If Section B students were penalized by a stricter interpretation, a documented remedy may be warranted; if the difference persists after calibration, the grades may stand. Either way, the decision needs a written rationale because it will set precedent for the third section next year.

This term: run a two-hour calibration session using four anchor recordings, two high and two mid, and have both instructors score independently; discuss discrepancies against behavioral descriptors. Blind double-score a stratified 20% sample of both sections, about 12 recordings, and calculate agreement. If the audit shows a systematic standard difference, re-score the affected Section B recordings using the calibrated standard with a second rater, and document the basis for any grade change. Respond to the appeal in writing with the evidence and the remedy. Next year at 90 students: rewrite the rubric with observable anchors, such as 'reflects feeling and checks accuracy with a follow-up question'; require rater training before grading; use a moderation sample of 15 to 20% with adjudication; and build role-play recordings into the LMS so scoring can be asynchronous. Do not rely on AI scoring unless validity is established. Assign a faculty assessment lead with workload recognition. The goal is not perfect objectivity but defensible, consistent judgment. Report reliability data to the curriculum committee and use it in faculty development.

Distinguish reliability from validity. Role-play assessment is judgment-based, as it is in clinical supervision, but judgment can be standardized through defined criteria, rater training, calibration, multiple raters, and moderation. The fact that two instructors applied different standards is evidence of poor assessment practice, not proof that the construct cannot be assessed. The chair should say this plainly to faculty and to the student: the program will not abandon performance assessment, because interviewing skill cannot be measured well by a multiple-choice test, but it will make the scoring process more consistent. At the same time, the chair should avoid promising that all subjectivity can be eliminated. Some variation in student performance is real. The response should separate three questions: whether this student was treated unfairly, whether the rubric is adequate, and whether the program has a reliable scoring system. The first may require a remedy; the second and third require revision and training. Communicating that distinction protects both the student's right to fair process and the faculty's professional judgment.