Calibrating a Writing Rubric with Two Raters
Two German teachers score a timed 120-word writing sample for a Seal of Biliteracy practice round. The prompt asks students to describe 'Mein Traumberuf.' One sample, by a non-heritage German 4 student, has strong content and varied vocabulary: 'Architektin,' 'nachhaltige Stadtplanung,' 'Barrierefreiheit.' It also contains systematic case errors after prepositions, several subordinate-clause verb-second errors ('weil ich will ...'), and one English-influenced structure ('Ich habe 18 Jahre' instead of 'Ich bin 18 Jahre alt'). The holistic rubric has four bands: text type, comprehensibility, vocabulary, and language control. Rater A assigns Advanced Low; Rater B assigns Intermediate High. The school requires a single score for the seal application and asks teachers to document evidence.
1. What score is most defensible, and how should the two raters document their evidence?
Read the worked answer
Intermediate High is the more defensible holistic score because the grammar errors are systematic in basic structures, not confined to complex sentences. The sample is coherent and paragraph-like, and the vocabulary is a genuine strength. But case after prepositions and verb-final order in subordinate clauses are core German structures; Advanced Low descriptors generally expect good control of basic structures with errors appearing mainly when the writer reaches for complexity. The text also remains descriptive rather than demonstrating sustained narration across time frames, which is often part of Advanced-level writing. Rater A likely overweights vocabulary and content. To document the decision, each rater should annotate the sample against all four bands: text type (paragraph, organized), comprehensibility (high despite errors), vocabulary (strong, precise), language control (systematic basic errors). The final score should include a short evidence note such as 'coherent paragraph and strong lexicon, but recurring case and verb-final errors.' If the school treats the seal as high-stakes, multiple samples should be used before a final decision.
2. What instructional feedback should follow from this scoring, and how should the teachers calibrate future ratings?
Read the worked answer
Feedback should target two high-leverage patterns: case after prepositions and verb-final order in subordinate clauses. Rather than marking every error, the teacher can give a model paragraph and ask the student to color-code prepositional phrases and underline each verb in a subordinate clause. Then use sentence-combining practice with 'weil,' 'dass,' and 'obwohl,' followed by a short rewrite of the same prompt. Address 'Ich habe 18 Jahre' as a lexical-grammatical chunk ('Ich bin 18 Jahre alt') because it is a common transfer error. For calibration, the two teachers should score three anchor papers together and discuss where the descriptors overlap: vocabulary strength should not cancel systematic basic errors, and comprehensibility should not be treated as grammar accuracy. They should record the evidence behind each score. A second timed writing sample later in the year would show whether the focused instruction transferred.