Common Portfolio Scores That Do Not Agree
A district moves Art I to standards-based grading. Four teachers share 140 students in five sections. The common summative portfolio asks students to submit three works and a written artist statement. The rubric has four criteria (composition, craftsmanship, color/value, reflection) and four levels. After quarter one, scores on craftsmanship range from 2.1 to 3.4 across teachers; the same portfolio scored by two teachers differs by one level 40 percent of the time. English learners score lower on reflection even when their visual work is strong. One teacher gives 4s for effort and participation. The department meets to revise the system before the next quarter; the principal wants defensible data and fewer compliance complaints.
1. Identify the main validity and reliability problems in the current portfolio system and propose a calibration process.
Read the worked answer
The largest problem is construct-irrelevant variance: effort and participation are being folded into a rubric that claims to measure art learning, and English proficiency is depressing reflection scores. The craftsmanship range across teachers suggests inconsistent definitions or severity, while the 40 percent one-level disagreement indicates rater drift and unclear level descriptors. A single portfolio of three works also gives a narrow sample, and teachers may be scoring presentation or compliance rather than learning.
Calibration should include anchor exemplars for each criterion and level, scored collaboratively and annotated with reasons. Teachers should score a sample of portfolios blind, in pairs, and discuss disagreements before scoring their own students. Use two raters for borderline portfolios and report the range, not only the average. Revise the rubric so each level describes observable evidence, such as 'edges are intentional and consistent' rather than 'good craftsmanship.' Remove effort and participation from the academic score. Repeat calibration at midyear and keep a record of decisions so the principal sees a defensible system.
2. Revise the assessment so English learners and students with language needs are not penalized twice.
Read the worked answer
Separate visual art criteria from language criteria. Reflection should be graded on the quality of artistic reasoning: identification of a choice, evidence from the work, and explanation of revision. Accept an oral recording, a scribed response, labeled sketches, or a bilingual statement. Provide sentence frames and a visual glossary, and translate the rubric into home languages when possible. Do not lower the visual criteria because the written English is limited, and do not give full credit for a generic statement that could describe any artwork.
Also build multiple measures: a short conference, a planning sheet, and an artist statement. For English learners, allow home-language notes during planning and a first draft in either language. The teacher can assess the same standard through a conference if writing is the barrier. This keeps expectations high while removing the language load that is not the target of the standard.
3. How should the department use the data instructionally and report it to the principal without turning it into teacher rankings?
Read the worked answer
Disaggregate by criterion, not just by teacher or overall average. If composition scores are strong but reflection is weak across sections, the department can plan a common formative check on reflection rather than blaming a teacher. Compare distributions and student work samples, and examine whether English learners are overrepresented at the lower reflection levels. Use the data to select professional learning, such as a calibration session or a workshop on language supports.
Report medians, ranges, and sample sizes, and explain the calibration process. Avoid publishing teacher-by-teacher averages, which encourages score inflation and does not improve validity. Instead, report the percentage of students meeting each standard, the reliability evidence from double scoring, and the instructional actions taken. The principal gets defensible data, and the department gets a cycle of inquiry rather than a compliance exercise.