Calibrating a Capstone Design Rubric Across Two Teachers
Two teachers co-teach a capstone engineering design course and share a four-criterion rubric - problem definition, design justification, prototype evidence, and communication - each scored 1–4. After scoring 27 portfolios independently, their means differ by 1.5 points on design justification, and they disagree by two points on six portfolios. One teacher tends to reward polished CAD renderings; the other rewards evidence of testing even when the writing is weak. An English learner's portfolio contains strong iteration photos and a labeled sketch but short, error-heavy text. A student with an IEP for written expression submitted a recorded oral defense instead of a written report. Scores are due in two days, and the state accountability system uses final grades from the course. The teachers must reconcile their scores and decide whether to revise the rubric or the evidence requirements before the next term.
1. Describe a defensible process for reconciling the two teachers' scores within the two-day deadline, and explain what should not be done.
Read the worked answer
Diagnose the disagreement before adjusting numbers. A 1.5-point mean gap on one criterion, plus six portfolios two points apart, indicates rater drift on the construct, not random error. The pattern suggests one teacher is scoring representation quality (CAD polish) while the other is scoring evidence of testing. Both are looking at real student work, but only one is scoring design justification.
Process that fits two days:
- Rewrite the design-justification descriptor in observable terms: cites test data, explains a trade-off between criteria, justifies a revision or a decision not to revise.
- Blind-score eight anchor portfolios together, including the six contested ones, and discuss until agreement.
- Re-score only the contested portfolios and any portfolio within one point of a grade boundary; keep the rest.
- Record the consensus score and a one-line rationale for each change.
Do not simply average the two scores; averaging preserves the construct confusion. Do not default to the higher score without evidence.
Separate the criteria. If communication is a distinct criterion, CAD polish belongs there. Design justification should be readable from a test log even when the drawing is rough.
Document the moderation so the department can defend the grades, since the state uses course grades.
2. How should the two teachers handle the English learner's portfolio and the IEP student's recorded defense so that the scores remain valid?
Read the worked answer
Name the threat to validity. The English learner's text is short and error-heavy, but the portfolio contains iteration photos and a labeled sketch - direct evidence of design justification. Scoring that criterion down because of language conflates communication with reasoning and introduces construct-irrelevant variance. The fix is to elicit the same evidence in another mode: a short oral interview using the photos as prompts, scored against the same descriptor.
The IEP student's recorded defense is not a favor; it is an accommodation already on the plan. If the course standard assesses engineering reasoning, the recording is valid evidence and should be scored with the same rubric. If communication is itself a standard, the student still needs a supported route to show it - a labeled diagram plus oral explanation, with the accommodation documented.
Boundaries: the teacher cannot invent accommodations after the fact without the team, and cannot waive a standard silently. Document what was provided and why, keep the evidence, and ensure the transcript grade means the same thing for every student.
Practical step: offer an oral-defense option to all students, which removes the stigma and makes the evidence mode a design choice rather than an exception.
3. Recommend revisions to the rubric and evidence requirements for the next term, and explain how the state use of course grades bears on those decisions.
Read the worked answer
Revise both the rubric and the evidence requirements, because the disagreement exposed a design flaw in the assessment.
- Split the constructs. Five criteria: problem framing, design justification, prototype evidence, iteration and reflection, and communication. CAD quality scores only under communication.
- Write anchored descriptors at each level with observable language - for example, level 4 justification cites specific test data, explains a trade-off between at least two criteria, and explains why an alternative was rejected.
- Standardize evidence. Require a test log, a decision matrix, and dated notebook entries from all students, so justification evidence exists regardless of writing fluency or drawing skill.
- Calibrate before grading, not after. Score anchor portfolios together at the start of the term and double-score a 20 percent random sample.
Accountability: since the state uses course grades, inflation is not a harmless kindness. Moderation records and common evidence are the defense.
Next-term plan: pilot the revised rubric on the first project, review disagreements at midterm, and keep the oral-defense option. Re-score the six contested portfolios under the new descriptors and compare results, which gives the department its own validity evidence.