Six Traits, Four Teachers, 23% Agreement
Four ninth-grade English teachers scored 140 argumentative essays on a six-trait analytic rubric (ideas, organization, voice, word choice, sentence fluency, conventions), each trait on a 1-4 scale, to produce a district writing sample. Each paper was scored once, by the teacher who taught the student. A moderation meeting reveals that exact agreement between any two teachers on total scores is 23%, that one teacher averages 0.8 points higher on every trait, and that the largest spread - 12 points out of a possible 24 - falls on a paper by an English learner whose essay takes a clear position with strong evidence but frequently drops articles and shifts to present tense when narrating past events. The district requires both that the scores be reported to families and that they be used to identify students for honors English. Two teachers argue the ELL paper must score 2 in conventions because colleges will judge the grammar. The department has 90 minutes for one more meeting before scores are due.
1. Diagnose the reliability problem and propose fixes that fit in 90 minutes.
Read the worked answer
The rubric is not the main problem; the scoring design is. Three flaws compound: each paper was scored by a single rater, that rater was the student's own teacher, and no calibration occurred before scoring. A single rater with a known stake (their own students' results reflecting on them) produces halo effects and score drift, and 23% exact agreement across a 7-25 point total range is barely better than chance.
What 90 minutes can actually fix:
- Calibration (30 min). Score five anchor papers spanning the range independently, then discuss. Differences usually shrink sharply once raters see what a 3 looks like in this building.
- Blindness (10 min). Remove names and replace with ID numbers before any further scoring. This alone attacks the biggest bias source.
- Trait-by-trait scoring (ongoing). Score one trait across all papers before moving to the next, rather than one paper across all traits. This reduces halo and is faster than it sounds.
- Double-score the decisions that matter (40 min). Have a 20% sample - 28 papers - scored by a second reader and compute agreement. If the honors cut score falls near a student's total, that paper gets a second read regardless.
What cannot be fixed today should be stated plainly: scores are adequate for descriptive reporting to families with a caveat, and not adequate for a high-stakes honors decision. Schedule the next window with the fixes already built in.
2. The ELL paper: should conventions be scored 2? Justify your decision from the rubric, not from consequences.
Read the worked answer
Start with what the trait is defined to measure. If conventions means control of grade-level edited American English, the paper legitimately scores below a 4 - that is what the trait is for, and refusing to score it lower would be its own kind of bias. The question is which lower score, and the answer is decided by the descriptors.
Most rubrics place a 2 at 'errors interfere with meaning or readability' and a 3 at 'errors present but do not impede understanding.' Dropped articles and inconsistent tense in a clear, well-evidenced argument do not impede understanding for a normal reader. That is not a 2. A 3 fits the descriptor.
The colleges argument is a consequence argument, not a scoring argument. It also conflates two different purposes: a district writing sample describes current performance against a rubric; predicting college outcomes is a separate claim that a single conventions score cannot carry. If the district wants a college-readiness signal, it should build one deliberately, on multiple samples, and report it separately rather than smuggling it into a trait score.
Two further safeguards. First, score conventions from a clean read of the paper, after the ideas score is finalized, so a lower conventions score does not drag down organization and ideas - the classic halo pattern with multilingual writers. Second, look for pattern rather than counting errors: are the errors systematic features of a developing second-language writer, or random slips, or signs the student was reaching for complex syntax beyond current control? That distinction belongs in the comment to the student even when it does not change the number.
3. Can these scores identify students for honors English? Answer with the specific decision rule you would apply.
Read the worked answer
No - not in their current form, and the reason is arithmetic rather than philosophical. With 23% exact agreement and no second reads, the band of plausible scores for any given paper is wide enough that students near any cut score would be sorted by which teacher they happened to have. Since one teacher runs 0.8 points high on every trait, that teacher's students would be systematically over-identified - a bias traceable to assignment, not writing.
A workable rule:
- Do not use the current totals for honors placement. Report them to families as descriptive, with a one-line note that scoring is being standardized.
- For this cycle, use a second, independent source for honors: a common on-demand prompt scored blind by two readers who did not teach the student, plus a portfolio of two revised pieces. Require agreement within one point; send disagreements to a third read.
- Publish the cut score and the process in advance so placement is not revisable after results are known.
- Track the identification rate by subgroup and by teacher at the next window. If one teacher's students are over-represented at the cut, that is a rater-training problem to fix before the next round, not a finding about the students.
The honest framing for the department: the writing sample is currently good for starting conversations about instruction and inadequate for sorting children. Say that to families in plain language, and the reporting requirement is still met.