Validity of a District ASL 1 Final Exam
A district with 300 ASL students is adopting a common final exam for ASL 1 and 2. The proposed exam has three parts: 40 English multiple-choice questions about ASL grammar, such as which sentence is topicalized; 10 picture-matching items in which students choose the English word for a sign; and a 3-minute voice-on interview in English about Deaf culture. The exam counts for 25 percent of the course grade. The district says it needs objective, comparable data across schools. The ASL teachers object, and a Deaf teacher on the committee says the exam measures English and test-taking skill, not ASL ability. The committee must vote in one month. The district has no budget for commercial ASL proficiency tests, but it does have video cameras, a learning management system, and two Deaf community members willing to help score.
1. Analyze the proposed exam for construct validity, reliability, and fairness. Identify the most serious threats.
Read the worked answer
The most serious threat is construct underrepresentation. The exam measures English reading about ASL grammar, English-to-sign recognition, and spoken English in an interview. ASL 1 and 2 ability includes receptive and expressive signed communication, which the exam barely samples. The grammar questions may measure metalinguistic knowledge in English, not the ability to use ASL. The voice-on interview measures English proficiency, which is not the target construct and may disadvantage deaf students or strong ASL students with weaker English. Reliability is not a substitute for validity: a consistently wrong measure is still wrong. Fairness is also threatened because the exam's English load and voice requirement may create construct-irrelevant variance. A 25 percent grade weight makes the consequences high. The committee should reject the exam as a sole measure, not merely tweak items.
2. Propose a balanced assessment plan for a 300-student program that includes at least one performance task and one receptive task. Explain how to keep scoring reliable.
Read the worked answer
A balanced plan for 300 students: (1) expressive performance task-students record a 2-3 minute signed narrative or dialogue with a partner, scored on a 4-point rubric for comprehensibility, grammar, space and classifiers, and discourse; (2) receptive task-students watch three short videos with different signers and answer comprehension questions in ASL or English, including one regional variant; (3) vocabulary in context-students identify signs in sentences rather than isolated pictures. To keep scoring reliable, train scorers with anchor videos, double-score a random 20 percent, hold calibration meetings, and report inter-rater agreement. Use the learning management system for video submission and the Deaf community members for scoring support. Keep the written grammar section as a minor component only if it is labeled as metalinguistic knowledge, not ASL proficiency.
3. How should the committee address the district's demand for objective data?
Read the worked answer
The committee should redefine objective data as reliable evidence about the intended construct, not multiple-choice format. A performance assessment can produce objective scores if the rubric is explicit, scorers are trained, and agreement is reported. The committee can offer the district: common rubrics, anchor performances, inter-rater reliability statistics, and growth data from pre- and post-recordings. It can also keep a short, low-stakes written section for program evaluation while excluding it from the proficiency grade. The key argument is that comparability across schools requires a common construct and common scoring, not a test that measures English. If the district insists on a single multiple-choice exam, the committee should ask for a validity claim and pilot data before attaching 25 percent of the grade.