Rater Disagreement on a Constructed-Response Essay
You are scoring the Italian IV constructed-response section: 32 persuasive essays on 'Le sfide dell'immigrazione in Italia,' each 200–250 words, using a 6-point analytic rubric with three criteria: content, organization, and language. Two raters independently scored all essays. For one paper, Rater A assigns 2 because the student writes 'mo' and 'aggio' and once omits an article; Rater B assigns 4 because the thesis is clear, paragraphs are coherent, and the student uses transitions such as 'd'altra parte' and 'innanzitutto.' The student's family is from Rome; the essay includes one invented spelling, 'immigrazzione.' The rubric says language is scored for 'control of standard Italian grammar and vocabulary' but does not define dialect or register. The department requires inter-rater agreement within one point on 90% of papers. You are the scoring coordinator.
1. As coordinator, how would you adjudicate this paper?
Read the worked answer
Rescore by criterion rather than split the difference. Rater A's 2 overweights two Roman dialect forms and one missing article; 'mo' and 'aggio' are not automatically failed standard grammar, and the rubric must distinguish dialect/register from error. Rater B's 4 may underweight the formal register requirement. I would assign content 4 for a clear thesis and relevant support, organization 4 for coherent paragraphs and transitions, and language 3 because the essay is generally comprehensible but contains one invented spelling and some nonstandard register features. On an analytic total that is 11/18; if the department requires one holistic 6-point score, 3 is defensible with a written rationale. Document the criterion-level evidence, not just the final number.
2. What rubric clarification and rater training would reduce this disagreement?
Read the worked answer
Define the language criterion explicitly: control of standard Italian for a formal academic register; regional dialect forms are not errors in themselves, but register choices should be described separately from morphosyntactic and spelling errors; code-switching should not lower the content score unless it impedes communication. Add anchor papers at scores 2, 3, and 4 that show dialect features, register shifts, and spelling errors. Train raters to score one criterion at a time and to underline evidence. Run a blind second scoring of a sample and discuss discrepancies without averaging them away. The goal is not to make raters identical but to make their judgments defensible and consistent with the construct.
3. A rater insists that any dialect form must receive a 2 because the rubric says 'standard Italian.' How do you handle that?
Read the worked answer
Use the rubric's purpose and fairness logic. The task asks for a persuasive essay in an academic register; it does not ask students to erase regional identity. If dialect forms interfere with clarity or are inappropriate to the register, the language score can reflect that, but a blanket 2 treats a legitimate variety as a deficit and confuses dialect with error. Show the rater two anchor papers with the same dialect features but different control of syntax and spelling, and ask which rubric descriptors actually apply. If the rater still disagrees, the coordinator's criterion-level score stands, and the department should revise the rubric wording before the next administration.