Paired Role-Play and the Reliability Problem
A German II class of 28 students completes a paired role-play: one student is a tourist asking for directions to the Stadtmuseum; the other is a local giving directions. The teacher and a student teacher rate three criteria on a 4-point scale: fluency, accuracy, and task completion. After 14 pairs, the two raters differ by two or more points for four students, and inter-rater agreement is low. One pair rehearsed a script and sounded fluent but could not answer 'Wie lange dauert es?' Another student was accurate but silent for 12 seconds and scored low on fluency. One student used 'left' and 'right' in English. The department wants interpersonal speaking assessed twice per semester; the teacher has 50 minutes for the whole class and no individual booths.
1. Explain why the paired role-play produced unreliable scores and propose at least three changes to the task.
Read the worked answer
The task confounded several variables. Pair composition varied, so a strong student could carry a weaker partner; the roles were unequal, with the local doing more of the linguistic work; and one pair memorized a script, which demonstrated rehearsal rather than spontaneous interaction. The 12-second silence may have been planning time, but the fluency scale treated it as low ability. The English words 'left' and 'right' may reflect a strategic repair or a genuine gap, and the raters did not agree on how to score it. Finally, the raters heard different pairs and had no anchor samples, so their standards drifted.
Change the task in three ways. First, use an information-gap map so each partner has different information and must ask questions. Second, add a short spontaneous follow-up from the teacher, such as 'Und wenn die Straße gesperrt ist?' Third, rotate partners and use two or three brief scenarios rather than one long rehearsed one. These changes make interaction necessary rather than performable from memory.
2. Design a practical rating procedure for 28 students in 50 minutes that improves reliability, including who rates, what is recorded, and how L1 use is handled.
Read the worked answer
Divide the class into three stations: independent listening practice, a directions-map task, and the assessed role-play. The teacher rates live at the role-play station while the student teacher monitors the other stations. Use seven-minute rounds so all 14 pairs can be seen, with each pair performing one short scenario and one follow-up question. Record 60–90 seconds of audio for each pair on a tablet; this allows a second rating of borderline cases and creates evidence for feedback.
Rate two criteria rather than three overlapping ones: communicative task completion and language control. Fluency can be described within language control, but silence should be interpreted as planning time unless it prevents the exchange. Before rating, calibrate with two anchor recordings-one strong Intermediate Low and one approaching it-and discuss the descriptors. For L1 use, note it but do not automatically penalize it; distinguish a single borrowed word used to keep the interaction going from a sustained switch to English. Teach repair phrases such as 'Wie sagt man …?' and 'Können Sie das wiederholen?' so students have a German option. Have the student teacher second-rate only the four discrepant students and any borderline cases.
3. How should the teacher respond to the student who froze for 12 seconds and to the pair that memorized a script?
Read the worked answer
For the student who froze, first separate planning from inability. A 12-second pause at the start may mean the student was assembling a route in his head; the teacher should allow one clarification or prompt, such as 'Fangen Sie bei der Kirche an,' and score what the student does after the pause. Teach pause fillers and requests for repetition-'Also …', 'Moment', 'Wie bitte?'-so silence is not the only option. Do not retroactively raise the score without evidence, but do not let a single pause erase an otherwise accurate exchange. Give the student a short reattempt with a familiar map.
For the memorized pair, use a novel follow-up: 'Die Straße ist gesperrt. Was jetzt?' or 'Wie komme ich mit dem Bus?' If they cannot adapt, the score should reflect limited spontaneous interaction even if the rehearsed portion was fluent. Explain the distinction between rehearsal and interpersonal communication, and give them a new scenario in which they must ask for clarification. Keep the feedback private and framed as a next step, not as cheating. The goal is to reward the ability to negotiate meaning, not to punish preparation.