The Self-Correction That Made a Listening Item Unanswerable
The listening section of the SLLC entrance test has 25 items and contributes 25% of the total score. In a two-minute lecture on tidal turbines, the speaker says: 'The prototype generated 1.2 megawatts-sorry, 1.2 megawatt-hours per day-which was enough for about 800 homes.' Question 12 asks: 'How much energy did the prototype generate per day?' Options: A) 1.2 megawatts, B) 1.2 megawatt-hours, C) 800 megawatts, D) 800 megawatt-hours. The key is B. The recording is played twice. In the pilot, 29% chose A, 52% chose B, 11% chose C, and 8% chose D. The item-review group notes that the self-correction is rapid and that the first figure is a unit of power, not energy. Several high-scoring candidates chose A. The committee must decide whether the item is a valid test of listening, whether the self-correction should be re-recorded, and how to treat the 29% who chose A.
1. Is Item 12 a valid listening comprehension item? Decide and justify.
Read the worked answer
It can be valid if the self-correction is acoustically clear, because repairing a false start is a genuine listening skill. The speaker explicitly withdraws 'megawatts' and substitutes 'megawatt-hours', so the final answer is B. But the item currently tests a narrow discrimination between two similar-sounding units, and the first figure is repeated in the correct option's neighbourhood, which makes A a powerful lure. The 29% choosing A, including high scorers, suggests the item may be measuring attention to a fast repair under time pressure rather than comprehension of the lecture's content. I would retain the item only if a review of the audio confirms that the repair is clearly audible and that the pause is sufficient. If the repair is blurred or masked by the following clause, the item is unfair and should be dropped. In either case, the item should be revised before reuse so that the correct answer is supported by meaning, not by catching a single repair.
2. What is the construct-relevant difference between A and B, and how would you revise the item to reduce guessing?
Read the worked answer
A is a unit of power; B is a unit of energy, and the question asks about energy per day. A candidate who knows the difference can still choose A if they miss the repair or if they assume the first number is the intended answer. The construct-relevant difference is therefore not just 'which number did you hear' but 'did you follow the speaker's correction and interpret the units correctly'. To reduce guessing, re-record the speaker saying: 'The prototype generated 1.2 megawatt-hours per day-that is the energy produced over a day, not 1.2 megawatts, which would be its power rating.' This makes the correction explicit and gives candidates a second chance to process the distinction. Alternatively, change the question to 'What does the speaker correct in her description of the prototype?' with options about the unit, the figure, the number of homes, and the location. That tests repair explicitly rather than hiding it inside a numerical detail.
3. The 29% who chose A are mostly candidates who scored above 70% on the rest of the listening section. What does this imply, and what should the committee do?
Read the worked answer
When high-ability candidates choose a distractor, the item may be measuring something other than general listening proficiency-perhaps prior knowledge that 'megawatts' is the usual unit for turbine output, or a test-taking habit of trusting the first number. If the point-biserial is low or negative, the item is not discriminating in the intended direction and should be dropped. The committee should compute the discrimination index and compare it with the rest of the section. If the index is acceptable, the item may still be valid but difficult; if it is near zero, the item is a trap. Given that the item also has a plausible alternative reading, I would drop it from this administration, rescore the section out of 24, and replace it with an item that tests comprehension of the lecture's main claim about tidal energy. The report should note that the item was retired for construct-irrelevant difficulty, not because candidates lacked listening ability.