The Item That Rewards the Wrong Readers
A B1 reading paper uses a 400-word article on urban beekeeping and ten four-option multiple-choice items. It was trialled on 120 learners whose teachers had placed them at B1. Item 6 has a p-value of 0.86 and a discrimination index of −0.12: the strongest 30 learners chose distractor B (41%) more often than the key (35%). Item 8 can be answered correctly from the article's title alone. The paper must contain ten items, the live session is in three weeks, and there is no budget for a second trial. The coordinator asks you to decide what to change, and to justify each change.
1. Diagnose item 6. What most likely explains a negative discrimination index, and what do you check before altering anything?
Read the worked answer
A negative index means the item is working backwards: the best readers are the most likely to get it wrong. Check three things, in this order:
- The key. A mis-keyed option produces exactly this pattern - strong readers answer correctly and are marked wrong. Verify the answer sheet against the passage before anything else.
- The passage. Can distractor B be defended from the text? When 41% of the top group choose one distractor, a second defensible answer is more likely than a coincidence. Re-read the paragraph and look for the sentence a reader could use to justify B.
- The construct. Does the item need knowledge from outside the passage? If B is true in the real world and not contradicted by the article, readers with background knowledge will pick it, and the item is measuring general knowledge rather than comprehension.
Only after those three checks should you conclude that the item is simply confusing.
2. With three weeks and no re-trial, decide the fate of items 6 and 8. Justify your choice in terms of validity and reliability.
Read the worked answer
Item 6 cannot stay as it is. A negatively discriminating item actively subtracts information from the total score, depresses reliability, and will be contested in any post-exam review. Item 8 is a different fault: it is not wrong, it is empty - an item answerable from the title measures nothing beyond general knowledge and contributes no evidence about passage comprehension.
With no re-trial, the lower-risk route is minimal repair rather than replacement. Rewrite item 6's stem and the offending distractor so that B is clearly contradicted by a specific sentence, and move item 8's stem to a detail located in the middle of the text. Edited items keep their known passage location; brand-new items bring unknown difficulty into a paper nobody can re-trial. Then run a cognitive walkthrough with four or five colleagues, asking each to point to the sentence justifying their answer. That is not a trial, but it catches the two failure modes that produced the problem: two defensible answers and outside-knowledge answers.
3. The mean total score across the 120 learners is 8.3 out of 10. What does that add to your reading of the paper?
Read the worked answer
It suggests the paper is mistargeted for the decision it has to support. A mean of 83% with items like item 8 in the paper means most candidates are bunched at the top, where the test gives the least information. For a pass/fail decision you want maximum precision near the cut score, not near the ceiling.
Two cautions before acting. First, the trial sample's level is only as good as the teachers' placement - if the group was actually above B1, the paper may be correctly aimed. Second, a high mean is partly an artefact of the two faulty items, so fix those first and only then judge targeting. If the paper is genuinely too easy, the remedy is not longer texts but more items requiring inference across paragraphs rather than location of single sentences.