Rater Disagreement on a B2 Argument Essay
Two raters score 80 B2 writing scripts on a four-point analytic rubric: task response, organization, grammar, vocabulary. Script 37 is 210 words; the prompt asks candidates to argue for or against remote work. The prompt also requires candidates to take a clear position. The writer presents balanced advantages and disadvantages, uses accurate complex grammar and varied vocabulary, but never states a clear position. It contains three memorized transition phrases and one sentence copied from the prompt. Rater A gives 3,3,4,4; Rater B gives 2,2,3,3. The program requires raters to be within one point on each trait; otherwise a third rater adjudicates. The third rater is unavailable, and results are due tomorrow. The writing coordinator must resolve the score and decide whether the rubric needs clarification.
1. Which trait most clearly explains the score gap, and what score would you defend for task response?
Read the worked answer
The clearest gap is task response: Rater A gives 3, Rater B gives 2. The prompt requires a clear position, and the script gives a balanced survey without one. Under a typical B2 descriptor, a clear position is part of task achievement, so 2 is more defensible than 3.
Organization can remain 3 because the balanced structure is coherent, even though the missing stance weakens the conclusion. Grammar and vocabulary should not be pulled down just because the position is weak: the script shows accurate complex grammar and varied vocabulary, so 4 and 4 are defensible. My adjudicated profile is 2,3,4,4, for 13/16. The total is not an average of 12 because trait scores are not interchangeable.
2. How should memorized phrases and copied prompt language affect the scores?
Read the worked answer
Memorized transitions are not automatically a grammar or vocabulary error. They become a problem only if the rubric penalizes formulaic language or if they obscure meaning. One sentence copied from the prompt is normal task borrowing, not plagiarism, unless the rubric forbids it.
The bigger issue is task response: if the memorized frames replace a real position, they contribute to the missing stance. Do not double-penalize by lowering grammar and vocabulary for the same feature. Flag the script for rater discussion, not for misconduct.
3. What rubric clarification and rater training follow from this disagreement?
Read the worked answer
Clarify the task-response descriptors with benchmark scripts: 3 should require a clear position sustained across the response; 2 should describe a position that is implied or inconsistent. Add an organization descriptor that distinguishes balance from lack of stance.
Train raters to avoid halo effects: strong grammar should not raise task response, and weak task response should not lower grammar. Use double scoring on a sample and require adjudication within 24 hours. The third-rater shortage is an operational problem, not a reason to average incompatible scores.