The Listening Task That Rewards Guessing
A listening test plays a 90-second recorded announcement at natural speed, twice. There are 6 multiple-choice questions, each with 3 options. A learner, Dario, scores 5/6. On review, the teacher notices that for 4 of the 6 items, the correct option is the longest one, and for 2 items the correct option contains a word spoken in the recording ("platform," "delay"). Dario says he "didn't understand most of it" but "picked the long answers." The teacher must decide whether Dario's score reflects listening comprehension, and what to do before reporting the result.
1. Is Dario's 5/6 a valid measure of his listening comprehension? Explain the reasoning.
Read the worked answer
No - the score is confounded by test-wiseness, and the item design invites it.
Two design flaws are visible from the setup:
- Length cue: if the correct answer is the longest option in 4/6 items, a learner can score well without listening. That is a test-construction error, not a learner error.
- Word-matching cue: options that repeat a word from the audio reward surface matching, not comprehension.
Dario's own report ("didn't understand most of it" but used the long-answer strategy) is direct evidence that his score came from the cue. A score produced by a guessing strategy that the items reward cannot be treated as evidence of the construct. The result should be flagged and not reported as a comprehension level until the items are revised or a parallel form is used.
2. What should the teacher do with the result before it is reported?
Read the worked answer
Do not report 5/6 as a comprehension score. Re-measure or annotate.
Steps, in order:
- Revise the items. Balance option lengths and remove options that merely echo audio words. Make distractors plausible.
- Re-administer with the revised form (or a parallel recording of similar length and speed).
- If re-administration is impossible, report the score with an explicit caveat: "item analysis showed a length/word-match cue; this score is not a reliable comprehension estimate."
What not to do: subtract points for "guessing," or tell Dario he failed. The defect is in the instrument. Penalizing the learner for a flaw the test built in is unfair and would not improve the measure.
3. Dario understood little but scored high. What does this tell you about the difference between item difficulty and construct validity?
Read the worked answer
Difficulty and validity are independent. An item can be easy and still invalid.
- Difficulty is how many learners get it right.
- Validity is whether getting it right requires the target skill.
An item answered correctly by everyone because the answer is the longest option is an easy item and an invalid one. Conversely, a hard item can be valid if the only route to the answer is comprehension.
For this case: the 5/6 looks like high ability but measures test-taking strategy. The teacher's job is to ask, for each item, "what mental work produces the correct answer?" If the answer is "noticing length," the item does not belong on a listening test regardless of its pass rate.