The Conflicting Test Results
Ms. Fatima teaches Grade 10 mathematics. She administered a unit test with 20 multiple-choice questions and 5 open-ended problems. The test was designed to measure problem-solving skills. After scoring, she noticed that some students who performed well on multiple-choice questions scored poorly on open-ended problems, and vice versa. The test's internal consistency reliability (KR-20) was 0.65. She is concerned about the validity of the test and how to use the results to inform instruction. She has 40 students in her class.
1. What factors might explain the discrepancy in student performance between the two test formats, and what does this suggest about the test's validity and reliability?
Read the worked answer
Factors for discrepancy:
- Different constructs: Multiple-choice questions may test recall and procedural knowledge, while open-ended problems assess higher-order problem-solving and reasoning. Students may have strengths in one area but not the other.
- Test format bias: Some students may be test-wise in multiple-choice but struggle to articulate reasoning in open-ended formats.
- Scoring subjectivity: Open-ended problems may be scored inconsistently, affecting reliability.
- Content coverage: The multiple-choice and open-ended sections may not be equally aligned with the curriculum emphasis on problem-solving.
Validity and reliability:
- The KR-20 of 0.65 is moderate but below the commonly acceptable threshold of 0.70 for high-stakes decisions, suggesting some inconsistency.
- The discrepancy indicates the test may not be unidimensional; it may be measuring multiple constructs. This threatens construct validity if the intent was to measure problem-solving only.
- The test may have low reliability for the open-ended portion due to scoring subjectivity.
Implication: The test results should be interpreted cautiously. Ms. Fatima should consider using multiple sources of evidence and possibly revise the test to better align with problem-solving objectives.
2. How can Ms. Fatima use the test results to inform her instruction and improve future assessments?
Read the worked answer
Using results:
- Analyze item difficulty and discrimination: Identify which items were too easy or too hard, and which discriminated poorly. This can reveal misconceptions.
- Disaggregate by format: Look at class patterns: if many students struggled with open-ended problems, it suggests a need to emphasize reasoning and communication in instruction.
- Provide feedback: Give students specific feedback on their problem-solving processes, not just scores.
- Differentiate instruction: Group students based on needs, e.g., those who need more practice with multiple-choice (perhaps basic skills) and those who need support in explaining reasoning.
Improving future assessments:
- Align assessment with objectives: Ensure a balance of item types that match the intended learning outcomes.
- Improve reliability: Use rubrics with clear criteria for open-ended items, and consider training scorers or using moderation. Increase the number of items to improve reliability.
- Pilot test: Try out items on a small group to check clarity and difficulty before using them in a graded test.
- Use a table of specifications: Ensure content validity by mapping items to curriculum standards.
By using the results diagnostically and refining assessment practices, Ms. Fatima can enhance both teaching and assessment quality.