C1 or B2 on a script with six errors
Two raters independently score the same 240-word article from the STEP Writing paper. The task: write an article for the school magazine arguing for or against banning homework. The script has a clear position, three developed reasons, paragraphing, and a memorable closing line. Vocabulary is adequate but repetitive ('very important' three times); there are six errors in 240 words (articles, one tense slip, one comma splice) and one clause that loses the thread mid-sentence. Under the four-criterion scale (Task Fulfilment, Organisation, Range, Accuracy), Ana awards C1 and Ben awards B2. The pair must agree before results are released, and the candidate's university application depends on a C1. The coordinator has 20 minutes.
1. Locate the real disagreement. Which criterion is actually driving the split?
Read the worked answer
Rule out shared ground first Both raters, reading the same script, would have to agree on Task Fulfilment (clear position, three reasons, addressed to a magazine audience) and Organisation (paragraphing, closing line that lands). Those criteria are not where a two-band gap comes from.
Test the remaining two
- Accuracy. Six errors in 240 words is roughly 2.5% error density, and none of the six blocks meaning - a reader reconstructs every sentence first time. Most C1 descriptors require errors that do not impede communication; most B2 descriptors require errors that are noticeable but do not obscure meaning. This script sits on the C1 side of that line, though not comfortably.
- Range. 'Very important' three times, plus safe vocabulary throughout, is the weakest feature of the script. A rater who reads Range as the gatekeeper will land on B2 even while agreeing the writing communicates well.
Diagnosis: the split is a weighting disagreement. Ben is letting Range (and possibly a halo from the mid-sentence stumble) pull the whole profile down; Ana is letting Task Fulfilment pull it up. Neither has applied the criteria independently.
2. Resolve it in 20 minutes without letting the university stake influence the outcome.
Read the worked answer
Procedure
- Both raters re-score criterion by criterion, writing one line of evidence per criterion with a quotation. No overall band yet.
- Compare criterion by criterion. Expect convergence on three of four; the argument is now confined to one cell.
- Apply the scale's own rule for uneven profiles - most four-criterion scales specify whether a profile is governed by the weakest criterion or by a best-fit judgement. Apply whatever the scale says, not what feels fair.
- If the single cell still splits, refer to a third, senior rater. Do not average C1 and B2 into a band that describes neither.
Guardrails
- Do not tell the raters about the university application. If they already know, state on the record that the stake is irrelevant and that the evidence lines, not the outcome, will be reviewed.
- Record the final band with the evidence lines attached, so the decision is auditable if challenged.
Likely outcome on this script: C1 on Task Fulfilment, Organisation and Accuracy with a B2-leaning Range is a coherent profile, and most scales will place it at the lower end of C1 rather than mid-B2 - but the scale text, not this intuition, decides.
3. What systemic change would reduce this kind of drift next session?
Read the worked answer
Benchmark and calibrate
- Build a set of five anchor scripts spanning B2 to C1, each annotated with the scale language it exemplifies, and have every rater score them before the live session. Raters who diverge on the anchors are re-trained before they touch live scripts.
- Pay particular attention to anchor scripts with uneven profiles - strong task response, weak range. Those are exactly where raters invent their own weightings.
Make the scale do the work If the scale does not say how to combine criteria, that is the root cause. Add one sentence: e.g. 'Where criteria diverge, the band is determined by the two criteria most central to the task purpose.' Ambiguity in the instrument produces disagreement that looks like rater error.
Monitor
- Double-rate a random 10% and track exact-agreement and adjacent-agreement per criterion. A criterion that splits repeatedly is badly worded, not badly applied.
- Review rater pairs after each session; persistent outliers get coaching, not blame.