Score Drift in a Community Leadership Certificate
A community leadership certificate is assessed through a portfolio against a five-criterion rubric, each scored 1-4, with a pass mark of 14/20. Three part-time assessors mark independently. After the rubric descriptors were shortened 'to make them clearer', the overall pass rate rose from 68% to 91% across 40 participants. A sample audit finds that one portfolio received 2 from one assessor and 4 from another on 'evidence of reflection'; the average score at one delivery site is 17.8, compared with 13.9 at another. Learners have already been told they passed, and the certificate is used by a local employer as a hiring signal. The programme manager must decide what to do before results are released publicly and what to change for the next cohort.
1. Diagnose the main threats to assessment validity and reliability in this case. Distinguish between rubric wording, assessor judgement, and site effects.
Read the worked answer
Three threats should be separated. Rubric wording: shortening descriptors may have removed behavioural anchors, so 'evidence of reflection' can be read as 'has a reflection section' by one assessor and 'shows changed practice' by another. That is construct underrepresentation and increased construct-irrelevant variance. Assessor judgement: the 2-versus-4 discrepancy on the same portfolio shows severe leniency/severity or different interpretations; without moderation, scores are not comparable. Site effects: the 17.8 versus 13.9 averages may reflect real differences in teaching, different assessor mixes, or local grading cultures; if assessors are not blinded to site, bias is plausible. The jump from 68% to 91% is not evidence of improved learning by itself; it may be a standard shift. The pass mark of 14/20 also makes small score changes consequential. Public release would overstate precision and could mislead the employer using the certificate as a hiring signal.
2. Recommend a moderation process that can be completed within two weeks, including what to do about learners already told they passed.
Read the worked answer
Within two weeks: identify every portfolio within two points of the pass mark and every portfolio with a criterion discrepancy of two or more; double-mark those blind, then hold a calibration meeting using anchor portfolios at each level. Require assessors to record a one-sentence justification per criterion and resolve disagreements by consensus, with the programme manager as final arbiter. Recalculate results under the consensus rule and document every change. For learners already told they passed, do not quietly withdraw the result: check the organisation's appeals and records policy, consult leadership, and if a material error is found, communicate in writing with an explanation and a remedy. If the main problem is rubric ambiguity rather than individual error, a defensible option is to retain passes but annotate the cohort's results as under review, then fix the process. Public release should be delayed until moderation is complete.
3. What changes to the rubric and assessor training would you make for the next cohort, and how would you evaluate whether they worked?
Read the worked answer
Revise the rubric so each criterion has observable descriptors at each level, with exemplars, and separate 'evidence of reflection' from 'quality of reflection'. Reduce halo by scoring one criterion at a time and requiring evidence citations. Train assessors on anchor scripts, run a calibration exercise with a target agreement standard, and double-mark a sample in every cohort. Evaluate with percent exact agreement and adjacent agreement, a kappa statistic if sample size allows, score distributions by site, and a review of appeals. Also track whether the rubric predicts later performance or employer feedback, though that is a longer-term validity question. The test is not whether scores look nicer; it is whether two trained assessors can reach the same judgement from the same evidence and whether the criteria represent the intended learning.