Argument Rubric, Rater Disagreement, and the Five-Paragraph Template
An 11th-grade team scores 32 timed argument essays with a district rubric weighted 40% conventions, 30% organization, 20% evidence, and 10% claim. Two raters disagree by more than one level on 11 of 32 essays. The prompt asks students to 'take a position on a new cell-phone policy' after reading two short articles; many essays summarize both articles and conclude 'I think both sides are right.' Most students have had no explicit instruction in counterargument. The team wants to add a grammar mini-lesson and a five-paragraph template with 'first, second, third.' One 90-minute PD session is available before the district writing assessment in three weeks. The team's shared drive also contains an older rubric that weights ideas at 50%.
1. Identify the main validity problem in the rubric and explain how it distorts instruction.
Read the worked answer
The rubric underweights the construct it claims to measure. Argumentation depends on a defensible claim, relevant evidence, and reasoning that connects them; here claim is worth only 10% and evidence 20%, while conventions are worth 40%. A student who writes a tidy, grammatical summary that concludes 'both sides are right' can score well without staking out or defending a position. The 11 disagreements in 32 essays are a signal that raters are filling in vague criteria differently, especially on organization and evidence. The older rubric weighting ideas at 50% is closer to the argument construct but still needs criteria for reasoning and counterargument. Revise the weights to something like claim 20%, evidence 30%, reasoning and counterargument 30%, organization 10%, conventions 10%, then norm with anchor papers. Instructionally, the current rubric teaches students that correctness matters more than thinking, which is the opposite of the assessment's purpose.
2. Why might the five-paragraph template and grammar mini-lesson fail to improve the essays? What should replace them?
Read the worked answer
The template addresses format, but the visible problem is the absence of a position and of criteria for choosing between competing claims. 'First, second, third' can produce three paragraphs of summary with no argumentative work; it may even reward the 'both sides are right' move by making each side a body paragraph. The grammar mini-lesson targets conventions, which the revised rubric should reduce to 10%; it does not teach students how to evaluate evidence or address a counterclaim. Replace these with instruction in the rhetorical situation: audience, purpose, and stance; a claim that takes a position; criteria for evaluating the two articles' evidence; and a counterargument paragraph that concedes, rebuts, or qualifies. Use short mentor texts and a debate or quick-write to force a stance before drafting. Then use a revision checklist focused on warrant and counterargument, not on adding transitional words.
3. Outline a 90-minute PD session that addresses the scoring disagreement and the instructional problem.
Read the worked answer
First 20 minutes: raters independently score two anchor essays with the current rubric, then compare. Display the 11 disagreements and ask which criteria caused them; this makes the reliability problem concrete. Next 20 minutes: revise the rubric weights and write 3–4 plain-language descriptors for claim, evidence, reasoning/counterargument, and organization. Third 20 minutes: norm on one essay together, then independently score a second and check agreement; require evidence citations for each score. Final 30 minutes: plan instruction. Model a think-aloud that turns 'both sides are right' into a qualified claim, then map a three-week sequence: stance and criteria, evidence evaluation, counterargument, revision conferences, and a timed practice with the revised rubric. End with a commitment to score a common set before the district assessment so disagreement is caught early.