One Play, Two Choices, Six Items
A B2 listening paper plays a three-minute studio interview once at about 150 words per minute; candidates answer six true/false items. With 80 candidates the mean was 4.6 out of 6, the standard deviation 0.9, and KR-20 reliability 0.48. In post-test interviews several candidates said they had guessed, and one candidate who scored 6/6 said she had understood only the first minute. The audio, the six-item count and the single play are fixed by the syllabus. You are asked whether the results can be trusted and how to repair the task without changing those three constraints.
1. Correct for guessing. What proportion of the items did the group actually know, and how seriously should you take the candidate who scored 6/6?
Read the worked answer
On six true/false items, chance alone delivers 3 correct. Remove that floor and the group's true performance is (4.6 − 3) / (6 − 3) ≈ 0.53: roughly half of the items that were not free were actually answered from comprehension. The raw mean of 4.6 flatters the group considerably.
The 6/6 candidate is the same story. Under pure guessing, the probability of six correct is 0.5⁶, about 1 in 64. Across 80 candidates you would expect roughly one such score by chance alone - and she is one. That does not prove she guessed throughout, but her own report plus the arithmetic means her score cannot be treated as evidence of full comprehension. This is exactly the danger of true/false: it produces plausible-looking high scores from candidates who understood almost nothing.
2. Reliability is 0.48. Interpret that figure alongside the standard deviation rather than on its own.
Read the worked answer
0.48 on six items is low, but the more damning number is the standard deviation of 0.9. If every candidate answered entirely at random on a six-item true/false test, scores would still spread with a standard deviation of about 1.2 (the square root of 6 × 0.5 × 0.5). The observed spread is smaller than chance alone would produce, which implies that the estimated true-score variance is close to zero: after guessing is removed, the items are separating almost nobody.
So the test is not merely unreliable, it is uninformative. Do not over-interpret KR-20 itself here - short tests are always penalised by it, and six items is very short. Read the coefficient together with the guessing floor and the SD; all three point the same way.
3. Redesign the task keeping the same audio, six items and one play. What format do you choose and why?
Read the worked answer
Change the response format, since that is what is broken - the audio and the single play are not the problem.
- Three-option multiple choice cuts the guessing floor from 50% to 33% per item without changing the listening demand or the item count. It is the smallest change with the largest immediate gain.
- Note completion or matching removes the guessing floor almost entirely, because a guess has to be a plausible word from the text. It is the better option if the syllabus permits it, but it raises the marking burden and introduces spelling decisions.
Whichever you choose: order the items as the information occurs in the audio, never test the same stretch of text twice, avoid negative stems, and make sure the first item does not sit in the first ten seconds. Then pilot on 15–20 learners. Even a small pilot will show whether the items now spread candidates; without one you are guessing about guessing.