Listening Scores with a Noisy First Minute
At a Georgian university language center, 48 B2 candidates take a 20-minute listening exit test. Section A has 10 multiple-choice items on a three-minute monologue recorded at 165 words per minute and played once. Section B has five items on a 140-word-per-minute dialogue played twice. Section A mean is 4.1/10; Section B mean is 8.8/10. Candidates with the strongest reading scores also perform poorly on Section A. In post-test interviews, several report a ventilation hum during the first 20 seconds and missing the topic; items 3 and 7 require exact number recall; item 8 has two defensible answers. One teacher proposes deleting Section A, another proposes replaying all audio twice. The coordinator must decide within one week, before scores are released, and must preserve comparability with previous cohorts.
1. What evidence would you use to decide whether Section A is measuring listening ability or construct-irrelevant noise? State a decision rule.
Read the worked answer
Reasoning:
- Separate construct from noise by triangulating evidence. Check item-level data: if high-reading and high-listening candidates miss items 3 and 7 disproportionately, and if item 8 has two defensible keys, those are item flaws rather than evidence of poor listening.
- Audit the audio to identify exactly which items depend on the first 20 seconds and confirm the hum.
- Compare the correlation between Section A and Section B after excluding flawed items. A strong positive correlation supports a listening interpretation; a near-zero correlation plus interview reports supports construct-irrelevant noise.
- Decision rule: retain Section A only if the audio is intelligible and the retained items discriminate in the expected direction. Exclude flawed items from scoring for this cohort and document the deviation. Do not delete the entire section, because that removes listening evidence and creates a comparability problem.
2. Recommend immediate scoring and administration changes for this cohort and for the next cohort, respecting comparability.
Read the worked answer
For this cohort, do not replay Section A now: a mid-session replay would create unequal conditions and invalidate comparability. Rescore with item 8 removed, and remove any item that depends on the inaudible first 20 seconds only if the audio review confirms the defect. Report the original and adjusted scores, flag the deviation, and use a documented equating or moderation procedure with previous cohorts.
For the next form, pre-test the audio in the actual room, run a sound check before every session, and field-test a two-play Section A rather than imposing it operationally; changing playback changes the construct. If Section A is retained, write items that test gist and inference rather than exact number recall. Items 3 and 7 should be revised, not merely replayed.
3. Identify one threat to fairness and one threat to reliability in the current plan.
Read the worked answer
Fairness threat: candidates seated near the ventilation source may have heard more hum, so the defect is not uniform across candidates. That gives some test takers an advantage unrelated to listening ability.
Reliability threat: item 8 has two defensible answers, which adds measurement error and makes scores depend on an arbitrary key. A further consistency risk is that the coordinator might delete a whole section for one cohort but not another. Document the item review, keep a chain of evidence, and apply the same decision rule to all candidates.