Inter-Rater Disagreement on a High School Portfolio Rubric
You teach Art II at a high school with 24 students. The final portfolio is scored by two teachers using a district rubric: 40 percent idea and creativity, 30 percent technique, 20 percent composition, and 10 percent reflection. Scores are on a 4-point scale and converted to a percentage. After the first scoring round, the two teachers differ by two or more points on 6 of the 24 portfolios. One student, Jordan, received an 89 from you and a 93 from your colleague and has appealed, arguing that the higher score is the 'real' one. Jordan's portfolio contains a technically strong but derivative series based on a well-known anime character, plus one ambitious but unresolved mixed-media piece. The district requires a defensible grade and a written response to the appeal within five school days. You have one common planning period before the deadline.
1. How would you analyze the disagreement and adjudicate the six discrepant portfolios?
Read the worked answer
A two-point discrepancy on a 4-point scale is large enough that the problem is probably in the rubric or the scoring process, not just in the students' work. I would first compare the two teachers' scores criterion by criterion. If the disagreement is concentrated in 'idea and creativity,' that is a warning that the descriptor is vague and subjective. If it is concentrated in 'technique,' we may be weighting polish over evidence.
With one planning period, I would run a short calibration, not a debate about Jordan. We would each independently score three anchor portfolios - one clearly proficient, one developing, one exemplary - and compare our reasoning against the rubric language. Then we would rescore the six discrepant portfolios blind to each other's marks. For any portfolio still discrepant, we would use a third reader or a consensus conference with evidence cited from the work. The rule would be: point to specific evidence in the portfolio, not to effort or likeability.
I would also check for a halo effect: a technically strong anime-derived series may score high on technique and low on originality, while an unresolved mixed-media piece may score high on idea and low on resolution. The rubric should allow that split. Adjudication should follow the evidence, and the written appeal response should explain which criteria were applied.
2. What is your response to Jordan's appeal, and how do you explain the grade without simply deferring to the higher score?
Read the worked answer
I would explain that an appeal is not a vote between two scores; it is a review of evidence against public criteria. The response should be specific: on technique, Jordan meets the standard for control and finish. On composition, the anime series is consistent but relies on a single borrowed template, so the evidence for original compositional decision-making is limited. On idea and creativity, the derivative series is coherent but does not demonstrate the risk-taking or synthesis described in the top descriptor, while the mixed-media piece shows ambition but is unresolved. On reflection, Jordan's statement describes process but does not analyze choices.
I would show Jordan the rubric descriptors and the evidence I used. I would not say 'the other teacher was wrong'; I would say that the consensus score is based on the full evidence and the descriptors. If the calibration process changed my judgment, I would revise the grade and say so. If it did not, I would explain the difference between technical skill and original concept, which is exactly what the 40/30 weighting is meant to capture.
I would also offer a path forward: a revision opportunity or a future assignment where Jordan can rework the mixed-media piece with a clearer resolution. The appeal response should be written, dated, and kept with the portfolio.
3. What changes would you make to the rubric and scoring process before the next portfolio cycle?
Read the worked answer
The first change is to make 'idea and creativity' observable. I would replace it with descriptors such as: generates multiple solutions, takes a purposeful risk, synthesizes sources rather than copying, and resolves an idea through material choices. I would include annotated anchor portfolios at each level so teachers share a referent. The anime-derived work is a good teaching case: it can be technically excellent and still score lower on synthesis if the student has not transformed the source.
The second change is procedural. Scores should be submitted independently before discussion, and any discrepancy above one point should trigger a consensus conference or third reader. I would build in a calibration session at the start of the semester, not during the appeal window. I would also separate 'technique' into control and intentionality so that polish is not automatically rewarded.
The third change is student-facing. I would give the rubric at the assignment launch, require process evidence such as thumbnails and revisions, and hold a short portfolio conference. That reduces the chance that a student experiences the grade as arbitrary and gives teachers more evidence than a final product alone.