Item Analysis and a Miskeyed Question on a Seventh-Grade Common Assessment
The seventh-grade ELA team has scored a 20-item common assessment on theme and central idea. The results show that 68 percent of students missed item 7, whose stem reads: 'Which statement best synthesizes the development of the theme across the passage?' The answer choices are long and include the phrase 'textual nuance.' Item 12 was marked correct on the key as B, but the team agrees that D is also defensible and perhaps better supported by the passage. Four teachers scored the constructed-response item using the same four-point rubric; double-scoring a sample showed exact agreement on only 55 percent of papers. Team members disagree about whether to enter the scores as they stand, curve the test, or reteach and reassess. Report cards close in one week.
1. What does the item 7 data most likely indicate, and how should the team distinguish a real skill gap from a problem with the item?
Read the worked answer
A 68 percent miss rate on one item is a signal to inspect the item before concluding that students cannot synthesize theme. The stem embeds 'synthesizes' and the choices include 'textual nuance,' which are academic vocabulary terms that may not be equally familiar to all students. If the assessed standard is analyzing theme development, then difficulty caused by the vocabulary of the question is construct-irrelevant variance: it lowers scores for reasons unrelated to the skill being measured.
The team should check several things. First, did students who missed item 7 also miss other theme items, or was item 7 an outlier? If performance on similar items was stronger, the item is suspect. Second, did the students who missed it disproportionately include English learners or students with IEPs? That pattern would suggest a language-load or bias problem. Third, do the answer choices have more than one defensible option? If so, the item is ambiguous.
The team should not simply reteach theme because one item performed poorly. Instead, it should separate the item's vocabulary demand from the theme demand. A quick re-teach of 'synthesize' and 'nuance' may be useful, but the more responsible step is to revise or replace item 7 and not count it as evidence of mastery. The item should be retained in the item bank only after revision and review.
2. What should the team do about item 12, which has a disputed key, and how should that decision be communicated?
Read the worked answer
When two options are defensible and the team agrees that D is at least as well supported as the keyed B, the item is not functioning as a reliable measure. The fair decision is to void item 12 for this administration rather than penalize students who chose D. The team should remove the item from the total and recalculate scores, or award credit to students who selected either B or D if the team believes both reflect the intended skill. Awarding credit to both is simpler and transparent, but it changes the maximum score; removing the item keeps the scale cleaner. Either approach must be applied consistently to every student.
The team should document the decision: the item number, the problem, the rationale, the date, and the revised scoring rule. The item should be flagged in the item bank and revised before reuse. Communication matters. Students and families should be told that one item was found to have more than one defensible answer and that scores were adjusted so no student was penalized. Teachers should avoid the appearance of changing scores to improve results; the explanation should focus on fairness and accuracy.
A curve is a different intervention and should not be used to mask a flawed item. If the team also believes the test was unusually difficult overall, that should be evaluated separately using the distribution and item difficulty data, not by applying an arbitrary curve.
3. How should the team address the low inter-rater agreement on the constructed-response item before report cards close?
Read the worked answer
Fifty-five percent exact agreement on a four-point rubric is too low to treat the scores as comparable across classes. The team should run a short calibration session before finalizing grades. Each teacher should independently score the same three or four anchor papers, then compare scores and discuss the reasoning. The goal is to clarify what a 2, 3, and 4 look like for each rubric trait-especially the trait that is causing disagreement, which may be evidence use or explanation of theme.
After calibration, teachers should re-score a sample of papers, particularly borderline ones, and compare again. If resources allow, every constructed response should be double-scored or at least 20 percent should be double-scored, with discrepancies resolved by discussion. The team should record the anchors and the clarified rubric language so future scoring is more consistent.
For the current reporting period, the team should use the calibrated scores rather than the original ones if the original scores were assigned before calibration and the differences are substantial. The constructed-response score should be reported separately from the multiple-choice score, or weighted in a way the team can defend. The team should also note that a single common assessment is not sufficient for placement or retention decisions. The longer-term fix is to build a shared bank of annotated anchor papers and to calibrate before each administration, not after.