What the Unit Test Actually Measures
A ninth-grade social studies team writes a 40-item unit test on the Industrial Revolution. The blueprint called for 30% recall, 45% application, and 25% analysis or evaluation. Item analysis from 112 students shows a KR-20 of .61 and a mean of 26 out of 40. Item 7 (p = .91, point-biserial = .05) asks students to name the year a factory act passed. Item 12 (p = .34, point-biserial = .48) is a document-based question requiring sourcing. Item 23 (p = .22, point-biserial = -.12) asks about a political cartoon the class never discussed. Items 28-34 are all vocabulary recall, with p values between .78 and .92. The team's average last year was 31 out of 40 on a different version. Two teachers want to curve the scores; one wants to drop the test. The principal asks whether the test 'shows what students know.'
1. Interpret the statistics for items 7, 12, and 23 and the KR-20. What does each imply?
Read the worked answer
Item 7 is easy and does not discriminate (p = .91, r = .05); it may still be legitimate if the blueprint includes recall, but it supplies little information. Item 12 is hard but strongly discriminating (p = .34, r = .48); high scorers get it and low scorers do not, which is what a document-based item should do, so it should not be dropped. Item 23 has negative discrimination (r = -.12); stronger students are more likely to miss it, which suggests a miskey, an ambiguous stem, or content never taught. That is a validity problem, not merely a difficulty problem. A KR-20 of .61 is below the .80 typically desired for decisions about individuals; because the test contains many very easy items and a narrow recall cluster, internal consistency is depressed, and the total score is not reliable enough to curve or to rank students.
2. Analyze the alignment and validity issues. Is curving or dropping the test appropriate?
Read the worked answer
The blueprint is not being followed: items 28-34 and item 7 are recall, so recall is overrepresented relative to 30%, while analysis and evaluation are thin. That is construct underrepresentation: the test cannot show what students can do with sources even if item 12 works well. Curving preserves a flawed score and hides the misalignment; dropping the test discards useful information about item 12 and about which objectives are weak. Neither is defensible. The principal's question should be answered as 'partly,' because the test measures vocabulary and some sourcing, not the full standard. A test with a .61 reliability and a misaligned blueprint cannot support high-stakes inferences, but it can support instruction if the results are read at the objective level.
3. Recommend revisions and a plan for using the results.
Read the worked answer
Revise item 23 after checking the key and the taught curriculum; either fix or replace it. Keep item 12 and write more items like it. Rebalance toward application and analysis, then pilot the revised form and recompute item statistics and reliability. Report the standard error so teachers do not overinterpret small differences. Use the current results formatively by objective: reteach the objectives behind the low items, and use item 12 as a model for instruction. Do not use this version for grades that count heavily; use it as a diagnostic. If the team wants comparable scores across years, it needs a stable blueprint and equating, not a different test each year.