Rater C and Candidate 17: Moderating a Speaking Assessment
Twenty-four candidates complete an 8-minute paired speaking assessment over two days. Three raters score four criteria - fluency, pronunciation, vocabulary, and task fulfilment - on a 5-point scale. After day one, Rater C has awarded fluency scores of 2 to six candidates whom Raters A and B scored 3 or 4. Rater C's pronunciation scores are close to the other raters, and her task-fulfilment scores are slightly higher. One candidate, No. 17, receives totals of 14, 15, and 11. The school has audio recordings for all candidates, a rubric with band descriptors, and one senior rater available for a third adjudication. The results must be submitted in 48 hours, and the board requires a written moderation note for any score changed by more than one band. The speaking section is 20% of the final grade.
1. Is Rater C's fluency pattern systematic severity or a legitimate difference in judgement? What evidence would you check before deciding?
Read the worked answer
It looks like criterion-specific severity, not general harshness. Rater C's pronunciation scores are close to A and B, and her task-fulfilment scores are slightly higher. If she were a globally harsh rater, all criteria would drift downward. The pattern is concentrated in fluency, which suggests she may be applying a narrower interpretation of the fluency band descriptors - for example, counting pauses or hesitation more strictly.
Evidence to check: (1) each rater's mean and range per criterion; (2) whether the six candidates share a feature, such as long pauses, repetition, or a particular first-language influence; (3) whether Rater C's written comments cite band descriptors or personal impressions; (4) a blind re-rating of two or three recorded candidates by all three raters. If C's fluency scores remain one to two bands below the consensus after blind re-rating, the pattern is systematic severity. If the differences disappear when raters hear the same recording without seeing names, it is a calibration problem, not a candidate problem.
2. Describe the moderation procedure you would use in the 48 hours available, including what to do with the six candidates and Candidate 17.
Read the worked answer
Procedure: (1) Hold a 30-minute calibration meeting using two anchor recordings, one mid-band and one borderline, and ask all three raters to score them blind. (2) Re-rate a 25% sample, including the six flagged candidates, from audio with names removed. (3) Compare criterion by criterion. If Rater C's fluency scores remain consistently one to two bands below the others, adjust those six scores to the consensus band and record the evidence. Do not simply average all three scores, because that would hide a systematic bias.
Candidate 17: the totals 14, 15, and 11 cannot be resolved by averaging to 13.33; the 11 is an outlier. Send the recording to the senior rater for blind adjudication, with the three original totals hidden. Use the adjudicated total as the final score. If the adjudicated score differs from any original by more than one band, write the required moderation note.
Documentation: record dates, raters, criteria, evidence, and decisions. Submit within 48 hours with a short note explaining the fluency adjustment and the Candidate 17 adjudication.
3. What should the written moderation note contain for any score changed by more than one band?
Read the worked answer
The note should contain: the candidate number and criterion affected; the original score from each rater; the final agreed score; the evidence used, such as recording timestamp and the specific band descriptor; the reason for the change; the adjudicator's name and role; and the date. For example: 'Candidate 17, overall total: original 14, 15, 11; final 15 after senior adjudication. Evidence: recording 2, minutes 3–6; fluency descriptor band 4 requires sustained speech with only minor hesitation; candidate met this for most of the task. The 11 was inconsistent with the other two ratings and with the recording.'
What to avoid: naming students in public documents, writing vague reasons such as 'felt too harsh', or changing scores without a descriptor reference. The note should be concise enough for an external moderator to follow the decision without hearing the recording. If a score is changed by more than one band, the note is mandatory; if by one band or less, a brief log entry is still good practice. Keep a copy with the mark sheet and send the summary to the board.