Rubric Drift Across Three Campuses
A vocational health programme is delivered at three campuses under a common curriculum and rubric. The external examiner reports 'inconsistent marking' in clinical case studies: mean marks are 68 at Campus A, 61 at Campus B and 58 at Campus C. The same rubric is used, but Campus C tutors interpret the 'clinical reasoning' criterion differently. The moderation sample is ten scripts per campus, reviewed by one internal moderator, and no calibration session has run for two years. Campus C teaches 42% of students and has a pass rate of 81%, compared with 93% at Campus A. Nine grade appeals have been lodged, mostly from Campus C. The programme leader argues that the differences simply reflect student cohorts, pointing to lower entry tariffs at Campus C. Student representatives say trust in marking is collapsing. The programme board needs a defensible way to investigate and respond.
1. How should the programme distinguish genuine cohort differences from assessment drift? Specify the evidence and method.
Read the worked answer
Treat the mean difference as a hypothesis, not a conclusion.
Step 1 - criterion-level analysis. Compare scores on each rubric criterion across campuses. If Campus C is lower mainly on clinical reasoning while knowledge and communication are similar, that points to interpretation drift rather than cohort ability. Step 2 - blind double marking. Select a stratified sample of 30 scripts per campus, including fails and borderlines, remove campus identifiers, and have two calibrated assessors from different campuses mark them independently. Calculate exact and adjacent agreement and differences by campus. Step 3 - cohort variables. Compare entry tariffs, English language scores, placement hours, supervision ratios and placement quality. If Campus C has lower entry tariffs, some difference is expected; but a 10-point gap with the same rubric still requires a criterion-level explanation. Step 4 - assessor interviews and moderation records. Ask Campus C tutors to explain how they apply clinical reasoning; review their feedback comments for evidence of different standards. Step 5 - item and grade analysis. Check whether the gap appears in all assessments or only coursework; check resit and appeal patterns.
The judgement is drift if blind marking reduces the campus gap or if criterion use differs; it is cohort difference if blind marking preserves the gap and cohort variables explain it. Either way, the programme needs calibration.
2. Recommend immediate and medium-term actions to restore assessment quality and student trust.
Read the worked answer
Immediate actions, next four weeks:
- Re-mark a stratified sample of Campus C scripts, including all fails and a random sample of passes, with an independent marker; correct any demonstrable marking errors through the appeals process.
- Run a mandatory calibration workshop for all assessors using the same 10 anonymised scripts; agree written descriptors for clinical reasoning with exemplars.
- Suspend the use of the current moderation sample until calibration is complete; replace single-moderator review with cross-campus pairs.
- Communicate to students what has been found, what is being re-marked and how to appeal; give a named contact and a timeline.
Medium term, next semester:
- Publish a rubric with annotated exemplars at pass, merit and distinction.
- Introduce double marking for all borderline and fail scripts across campuses.
- Train assessors annually and record completion.
- Report campus-level grade distributions to the programme board each semester with an equity commentary.
- Commission an external review of the clinical reasoning criterion.
Monitor appeal rate, grade changes after re-mark, inter-rater agreement and a student trust survey. Do not dismiss the gap as cohort difference before the blind marking evidence is in.