Two Teachers, One Rubric, A Twelve-Point Spread
Two seventh-grade teachers co-teach a writing course and have just scored the same 30 narrative essays. Ms. Aran gave 12 A's; Mr. Cole gave 4. Their rubric has six categories - focus, organization, elaboration, word choice, conventions, and voice - each on a 4-point scale, plus an overall impression score the teachers weight heavily. Comparing sheets, they agree within one point on conventions and organization but differ by two or more on elaboration and voice on 11 essays. Ms. Aran says she scores up for students who worked hard; Mr. Cole says he compares each essay to the best one he read. The department wants scores defensible to parents, and the next unit starts Monday.
1. Explain the disagreement in terms of scoring error. Which categories are most vulnerable, and why?
Read the worked answer
Three errors are operating at once.
- Leniency/severity: a stable rater tendency. Ms. Aran's 12 A's versus Mr. Cole's 4 is too large to be essay-to-essay variation on a shared set.
- Halo effect: Ms. Aran's "worked hard" comment shows a non-writing quality leaking into a writing score. Effort is not in the rubric, and letting it in inflates every category.
- Rater drift/comparative judgment: Mr. Cole scoring each essay against the single best paper converts a criterion into a ranking and depresses mid-range scores.
Why elaboration and voice. These are the least operationalized constructs. Organization and conventions have visible features - paragraph breaks, punctuation - so raters converge. "Elaboration" and especially "voice" can mean detail, length, personality, or enthusiasm; when descriptors are adjectives rather than criteria, each rater supplies a private definition. The 11-essay split is a construct problem as much as a rater problem.
2. Propose a defensible scoring process for the next unit. Address the rubric, the anchors, and the procedure.
Read the worked answer
Rubric. Rewrite descriptors as observable features with a quantity or quality threshold per level: for elaboration, "includes at least two specific details that advance the conflict" beats "well elaborated." Either define voice as identifiable choices - diction, sentence rhythm, pacing - or drop it; an undefinable category cannot be defended to a parent.
Anchors. Select two or three essays per score level, scored jointly and annotated with the specific phrases that justify the score. Raters then compare to the anchor, not to the best paper in the pile - which removes the comparative-judgment problem directly.
Procedure. (1) Score blind to names. (2) Calibrate on four common papers before independent scoring; discuss disagreements until the reasoning, not just the number, converges. (3) Score analytically, category by category, across all papers before moving to the next category - this reduces halo. (4) Any score differing by more than one point on any category goes to a third read.
Effort. Remove it from the summative score entirely; if the department wants to recognize effort, do it in a separate comment or a non-graded record.
3. What follows instructionally for "voice," given the disagreement?
Read the worked answer
Teach voice as craft, not temperament. If voice stays in the rubric, students need to know what produces it: choosing a concrete noun over a general one, varying sentence length for pace, letting a narrator's diction reveal attitude. Those are teachable and visible in a text. "Sounds like the student" is neither.
Use mentor sentences. Show two versions of the same moment - one flat, one with deliberate rhythm and word choice - and ask students which choices create the effect. This converts an aesthetic judgment into an analyzable one and gives both teachers a shared vocabulary for scoring it.
Separate voice from enthusiasm. Ms. Aran's "worked hard" reflex may be responding to something real - a student taking a risk - but rewarding it under voice teaches students that personality earns points. Name the actual feature in feedback ("the shift to short sentences in the last paragraph speeds up the ending") so the student knows what to repeat.
If the descriptors cannot be made observable before the next unit, the defensible move is to remove voice from the summative and teach it formatively until the department can agree on what it means.