Performance Descriptor Calibration on a Modelling Task
Three teachers independently mark the same set of 30 student responses to a modelling task worth 20 marks. The task asks students to model the cooling of a cup of coffee and interpret the constant k. The descriptor bands are: 'High' (thorough, accurate, well-justified), 'Medium' (sound with minor gaps), 'Low' (partial, significant gaps), 'Not shown'. Teacher A awards a mean of 14.2, Teacher B a mean of 11.8, Teacher C a mean of 13.0. On a sample of five scripts, A and B disagree by 3 or more marks on three of the five. Teacher A says 'I reward effort and clear working'; Teacher B says 'I only give marks for correct mathematics'. The panel must decide whether the descriptors are fit for purpose before finalising scores.
1. Analyse the source of the marking discrepancy and recommend a calibration process.
Read the worked answer
The discrepancy is not random noise; it reflects different interpretations of the descriptors. Teacher A interprets 'well-justified' and 'sound' as rewarding visible reasoning and effort, inflating scores for well-presented but mathematically incomplete work. Teacher B interprets the descriptors strictly as correctness of the mathematics, deflating scores for correct reasoning with minor errors. The descriptors themselves are too vague: 'thorough', 'sound' and 'partial' are not operationalised, so each teacher fills the gap with personal philosophy.
Recommended calibration:
- Rewrite each band with observable criteria: for example, 'High' requires a correctly specified model with justified assumptions, correct parameter interpretation and a stated limitation.
- Run a moderation session: all three teachers mark the same five scripts independently, then discuss each mark against the written criteria.
- Adjust the descriptors where disagreement persists, then re-mark a fresh sample.
- Record the agreed exemplars as a reference set for future tasks.
This process addresses validity (do the descriptors measure the intended outcome?) and reliability (do different markers agree?).
2. Teacher A argues that rewarding 'clear working' is required because communication is part of the outcome. Is this a valid justification for the higher mean? Explain, distinguishing communication from correctness.
Read the worked answer
Partly valid but misapplied. Communication is legitimately part of the outcome, so clear setting out, correct notation and coherent explanation should earn credit. However, communication credit must be defined and capped, not used as a general uplift. If Teacher A awards communication marks for neat presentation regardless of whether the model is correct, the marks conflate two constructs and inflate the mean.
The defensible approach is to separate the constructs explicitly: allocate a defined portion of the 20 marks to communication (for example 3-4 marks) with criteria such as correct notation, logical structure and clear interpretation, and allocate the remainder to modelling accuracy and justification. Then a student with clear working but a wrong model earns the communication marks and loses the modelling marks. Teacher A's justification supports including communication criteria, not raising the overall mean.
3. After moderation, the panel finds that the task itself may be flawed: the coffee-cooling context allows multiple valid models (Newton's law versus a linear approximation over a short interval). How should the panel treat responses that used the linear approximation?
Read the worked answer
The panel should treat the linear approximation as a legitimate modelling choice if the student justifies it - for example, by noting that over a short interval the cooling curve is approximately linear. The descriptor should reward the quality of justification, not conformity to a single expected model. However, the panel must check whether the linear model can answer the interpretation question about k: if k is defined only within Newton's law, a student using a linear model must define their own parameter and interpret it consistently. Responses that do so should be marked as meeting the modelling criteria; responses that use a linear approximation without acknowledging its limits should lose justification marks.
The panel should also record this as a task-design lesson: the prompt should either specify the model or explicitly invite justified alternatives, and the marking guide should list acceptable alternatives. This protects both validity and fairness.