Accent, speed or lexis: dissecting four collapsed items
The STEP Listening section is played once. After the November administration to 24 students, nine complain that Speaker 3 - a Scottish urban planner recorded in a café - made Items 11–14 impossible. The audio runs at about 210 words per minute and contains 'outwith', which no one in the class knew. Item statistics: p-values for 11–14 are .31, .28, .35 and .30, against a section mean of .61. In the top quartile by total listening score, the same four items average .55. The café background is audible but no key information is obscured; the transcript shows all four answers delivered in clear syntactic frames. The school wants the items dropped and the recording redone with a 'neutral' accent, and has asked you whether the section was unfair.
1. Is the difficulty of Items 11–14 construct-relevant? Argue from the statistics and the transcript.
Read the worked answer
Separate the candidate explanations
- Accent. If accent alone blocked comprehension, the top quartile would collapse to chance (.25). They average .55 against a section mean of .61 - depressed, not destroyed. Accent is a contributor, not the cause.
- Speed. 210 wpm is fast for an exam but within the range of natural broadcast speech; it affects all items after the passage begins, so check whether items 8–10 from the same speaker also dipped. If they didn't, speed is not the driver.
- Lexis. 'Outwith' is the one factor the class could not have prepared for. The decisive question is whether any key depends on it. If an item's answer hinges on that word, the item is measuring vocabulary breadth, not listening - that is construct under-representation and the item must go.
- Noise. Audible but non-masking is authentic and defensible; masking would not be.
Verdict: the section is not 'unfair' as a whole. At most one or two items are compromised, and only if the transcript shows the key turning on 'outwith'.
2. What do you advise the school to do, and what do you refuse to do?
Read the worked answer
Advise
- Pull only the items whose key depends on the low-frequency dialect word; credit or discard those, not the whole block.
- Retain the Scottish speaker. Removing every non-standard accent narrows the listening construct to 'comprehension of received pronunciation', which is not what the syllabus claims to test and would make the score less meaningful, not more.
- If the café bed audibly overlaps a key phrase, re-master the audio (reduce the bed under the target segment); that is a technical fix, not a construct change.
Refuse
- Re-recording the block in a 'neutral' accent for the same cohort, which would give those 24 candidates a different test from everyone else and make the scores non-comparable.
- Dropping items purely because students complained. Complaint volume measures surprise, not validity; the top-quartile data is the evidence that matters.
Communicate with the item statistics, not reassurances: show the school that the items were hard for everyone and that the top quartile still performed above chance.
3. Design a check that would tell you whether the accent or the lexical item caused the failure.
Read the worked answer
A two-cell comparison
- Build two short parallel extracts from the same speaker: Extract X keeps 'outwith' and the accent; Extract Y replaces 'outwith' with 'outside' and keeps everything else (accent, speed, café bed). Same four questions, same candidates, counterbalanced order.
- If scores rise sharply on Y, the lexical item is the cause. If Y is unchanged, accent/speed is doing the work.
Add a third cell to separate speed from accent: Extract Z re-records the same script at 160 wpm in the same accent. If Z lifts scores, pace is the lever.
What makes this a check rather than an experiment
- Use 20–30 candidates at the target level, not the 24 who complained (their expectations are now set).
- Pre-test any new item on a small group before it enters the bank.
- Record the outcome in the item metadata: accent, wpm, lexical load. Over two or three cycles you get a defensible difficulty budget instead of an argument.