A Unit Test That Tells Two Different Stories
Ms. Reyes teaches eleventh-grade U.S. history at a comprehensive high school. She gave a 42-item unit test - 34 selected-response items and 2 constructed-response items scored on a 4-point rubric - to 96 students across three sections. The item analysis shows 88 percent of students answered the vocabulary and timeline items correctly, but only 37 percent answered the four items that asked them to evaluate a primary-source excerpt. Each of the four source items pairs a 90-word excerpt with a single question. Four students in her second period are English learners at the expanding level. On the constructed-response items, she and her co-scorer, a colleague teaching the same course, differed by two or more points on 11 of 30 sampled papers; the rubric's top band reads 'shows strong understanding.' Quarter grades close Friday, and her principal has asked the history team to bring a plan to next week's PLC meeting.
1. What can and cannot be concluded from the item data? Identify at least two competing explanations for the 37 percent and describe what evidence would distinguish them.
Read the worked answer
What the data support: this class can recall vocabulary and chronology but performs poorly when asked to evaluate a source excerpt. That is a pattern across two very different cognitive demands, and it is the most defensible reading of the item analysis.
What the data do not support: the conclusion that students 'cannot analyze sources.' The four items share a 90-word excerpt written at grade level, so at least two competing explanations remain. First, students may lack the historical reasoning - sourcing, corroboration, contextualizing - because it was never modeled. Second, the items may carry construct-irrelevant difficulty: reading load and unfamiliar format rather than absent reasoning. A third possibility is that the items were poorly written, with implausible distractors.
Distinguishing evidence: give the same reasoning task with a shorter or scaffolded excerpt; disaggregate the four items by prior reading scores to see whether only weaker readers failed; and check lesson plans for whether source evaluation was actually taught and in what form. If strong readers also failed, the problem is instruction or item design; if only weaker readers failed, it is literacy access.
2. How should Ms. Reyes resolve the scoring disagreement in time for Friday's grade deadline?
Read the worked answer
Averaging two incompatible readings of the rubric would hide the problem rather than solve it, and it would make grades unreliable for 96 students. The defensible sequence, even in three days, is:
- Build an anchor set with her colleague - three papers at each score point, discussed and described in observable language ('cites two pieces of evidence and explains how each supports the claim').
- Rewrite the vague top band ('shows strong understanding') into concrete features that two readers can apply independently.
- Rescore a fresh sample blind, then compare. If disagreement of two or more points persists on more than a few papers, bring in a third reader or the department chair.
For the quarter grade she should grade only what she can defend: apply the anchored rubric, keep the scores that are justified, and if the prompt itself was flawed, address that at the class level rather than through individual requests. She should send her principal a short note documenting the process - not to justify a curve, but so that any parent question meets a record of consistent practice.
3. What should the history team bring to the PLC meeting, and what should they agree on before the next unit?
Read the worked answer
The team should bring four things. First, item-level data disaggregated by item type rather than a single average, so the gap in source evaluation is visible to everyone. Second, the revised rubric with anchor papers, so scoring becomes comparable across sections and future common assessments mean the same thing in every room. Third, a short blueprint revision: fewer recall items, more brief excerpts with brief scaffolds, and at least one item per standard so the test samples the standards rather than the textbook's chronology. Fourth, an access plan for English learners - shorter excerpts, glossed vocabulary that is not itself the assessed term, sentence frames for the written response - that preserves the reasoning demand.
They should also agree on two or three common formative checks during the next unit. A benchmark should confirm a pattern the team already suspected, not deliver the first evidence of the year. Agreeing in advance on what a proficient response looks like is the step most likely to keep the next round of scores comparable.