Calibrating Oral Proficiency Scores Without Compromising Standards
At a district-wide French team meeting, six teachers review results from a common French III presentational speaking assessment. Each of 140 students recorded a three-minute speech; two raters scored it on a four-category rubric (1–4 points). After the first round, 22% of paired scores differ by two or more points, especially on fluency and cultural appropriateness. One parent disputes a score of 2/4 and asks for a regrade. A coach asks a teacher to raise an athlete's score from 2 to 3 to preserve eligibility. A teacher also mentions a shared drive containing copied multiple-choice items from a publisher's teacher resource without a current license. The team has one 45-minute meeting before scores are finalized and must decide how to calibrate, respond ethically, and protect test security.
1. Analyze likely sources of the rater discrepancy and propose a calibration procedure the team can complete or begin in the 45-minute meeting.
Read the worked answer
A discrepancy of two or more points on 22% of pairs suggests unclear rubric language and inconsistent evidence collection, not necessarily bad faith. Fluency descriptors may mix speed, pause length, and comprehensibility; cultural appropriateness may be judged without anchoring what counts as evidence. Calibration should use anchor samples: the team scores two recorded speeches independently, compares, and rewrites the rubric descriptors with observable language, such as 'sustains simple sentences with pauses that do not block meaning.' Then raters score a third sample and discuss discrepancies against the revised descriptors. Set a decision rule: any pair differing by more than one point is adjudicated by a third reader. Document the process. This is more defensible than averaging scores blindly or letting one rater's holistic impression dominate.
2. Explain how you would respond to the parent's regrade request and to the coach's request to raise an athlete's score. Include the professional principles that guide each response.
Read the worked answer
To the parent: explain the rubric, provide the scored descriptors and the specific evidence from the recording, and offer a review by a second rater under the calibration rule. Do not change a score merely because of a complaint; changing it without evidence undermines validity. If the regrade finds an error, correct it and explain. To the coach: refuse to raise a score for eligibility. State that grades and proficiency scores must reflect evidence; offer legitimate support, such as practice or reassessment after instruction if policy allows, but not a false score. Keep the conversation confidential and document it. The professional obligation is to the integrity of the assessment and to equitable treatment of all 140 students, not to a single athlete's eligibility.
3. Recommend safeguards for the shared drive of copied publisher items and explain how test security relates to the validity of the common assessment.
Read the worked answer
Test security requires immediate action. Do not use the copied publisher items; locate the license or remove the folder. Report the shared drive to the department chair or administrator, because copied materials without a current license may violate copyright and district policy. If the items have already been used, document which students saw them and do not reuse those items for secure assessments. Replace them with locally written items aligned to the same standards and have a second teacher review them. Remind the team that sharing secure items across schools can invalidate scores. The ethical principle is simple: assessment materials must be legally acquired, access-controlled, and used only for their intended purpose; convenience does not justify compromising security.