Reading Items That Reward Test-Wiseness
At a regional exam board, a 60-item reading pilot is administered to 120 students. Four passages are followed by 15 items; the time limit is 55 minutes. Item 9 asks for the main idea; its correct option is the longest and repeats words from the first sentence. Item 9 has 82% correct, including many low scorers. Item 12 asks for an inference; only 18% answer correctly, and high scorers split between B and D. The passage for item 12 mentions 'Boxing Day' without explanation. Item 14 can be answered from the title alone. Twenty-two students leave the final passage blank. A reviewer calls the section 'too hard' and wants to lower the cut score. The board must decide which items to retain, revise, or drop before the operational form is built.
1. Classify the main validity threat for items 9, 12, and 14, and for the blank final passage.
Read the worked answer
Item 9 is a test-wiseness threat: the longest option and lexical overlap allow low scorers to answer without reading the passage. It measures option-matching more than main-idea comprehension.
Item 12 is a cultural-knowledge and key-ambiguity threat: 'Boxing Day' is unexplained, and high scorers split between B and D, so the intended inference may not be recoverable from the passage alone. Item 14 is passage-independent because the title gives the answer; it measures prior knowledge or guessing.
The blank final passage is a speededness threat: 22 of 120 students leaving it blank suggests time pressure, not necessarily low reading ability. Each threat needs a different remedy; lowering the cut score would mask all four problems.
2. What revision would you make to item 12, and what evidence would justify the new key?
Read the worked answer
Revise item 12 by replacing the unexplained cultural reference with information inside the passage or a brief gloss, and rewrite the stem so the inference depends on a stated contrast rather than outside knowledge. Then pilot the revised item with high-scoring students and inspect distractor rationale.
If B and D remain defensible after revision, drop the item rather than force a key. The new key needs evidence: high-ability students should choose it more often, each distractor should map to a plausible misreading, and the key should be justifiable from the passage alone. Do not simply change the key to the most popular high-scorer choice; popularity is not validity.
3. Should the cut score be lowered because 22 students left the final passage blank? Justify your decision.
Read the worked answer
No. A lower cut score would change the standard, not fix the measurement problem. First check whether the form is speeded: passage length, item count, and the 55-minute limit. If 22 students blanked the final passage because of time, shorten the passage or extend time for the operational form, then equate scores.
If the section is intentionally speeded, the rubric must say so and the score interpretation must change. Lowering the cut score without evidence would pass candidates who did not reach the final passage and would weaken the board's defensibility.