Automated essay scoring pilot at a tertiary provider
A tertiary provider is piloting an automated scoring model for 6,000 first-year writing assessments. The model was trained on 12,000 essays previously scored by human markers. Overall agreement with human markers is 89%, but analysis shows that for students with English as an additional language the model scores 0.7 bands lower on average, and 8% of Māori and Pacific students are incorrectly flagged “at risk”. The model uses essay length, vocabulary rarity, sentence complexity, and a small set of grammar features. The vendor calls 89% “industry-leading” and proposes using the model for provisional grades, with human review only for flagged students. The semester starts in six weeks.
1. Evaluate the vendor's 89% agreement figure. Explain what additional evidence you would require before accepting the model for provisional grades.
Read the worked answer
Eighty-nine percent agreement sounds high, but agreement is not accuracy. If 89% of essays agree within the same band, we still need to know the direction and size of disagreements. A model that is one band too low for a whole group is not benign; it systematically disadvantages those students. The group-level gaps are the more important finding: a 0.7-band average deficit for EAL students and an 8% false “at risk” rate for Māori and Pacific students indicate that the model has learned features correlated with language background rather than writing quality.
The feature set explains why. Essay length and vocabulary rarity are proxies for prior exposure to academic English, not necessarily for argument quality. Sentence complexity can penalise concise, clear writing. A model trained on 12,000 human-scored essays inherits whatever biases the human markers had, and may amplify them because it applies them consistently at scale.
Before accepting provisional grades, I would require: a confusion matrix by band, not just overall agreement; disaggregated error rates by language background, ethnicity, gender, and disability; calibration analysis (when the model says “B”, how often is it a B?); test-retest consistency; and a blinded human adjudication of a stratified sample of 300 essays, including disagreements in both directions. I would also ask for the training data's demographic composition and whether the 12,000 essays came from the same assessment tasks and student population. A model trained on different prompts may not transfer.
Finally, “provisional” matters. If students can appeal and a human makes the final decision, the risk is lower. If the provisional grade determines progression or support, the model is making a high-stakes decision and the evidence bar is much higher.
2. Recommend a mitigation strategy for the group-level bias. Explain why removing ethnicity from the feature set is not sufficient, and what you would do instead.
Read the worked answer
Removing ethnicity from the feature set does not remove bias, because the remaining features are correlated with ethnicity and language background. Essay length and vocabulary rarity are exactly the proxies that carry the disparity. A model can be “blind” to ethnicity and still produce ethnically disparate outcomes. This is the standard proxy problem: fairness at the input does not guarantee fairness at the output.
A better strategy has several layers. First, measure outcomes by group continuously and set explicit acceptance criteria, such as a maximum 0.2-band average difference and a maximum 2-percentage-point gap in false “at risk” flags. Second, change the modelling target: train on rubric dimension scores rather than a single holistic score, and include features that are less language-dependent, such as whether the essay addresses the prompt, uses evidence, and acknowledges counterarguments. Third, use group-specific calibration only if it is legally and ethically defensible and transparently governed; a simpler alternative is to widen the human-review band around the model's score for students whose predicted score is close to a grade boundary. Fourth, require human review for all “at risk” flags before any support action is taken, and give students a route to contest.
The governance matters as much as the model. The provider should not deploy the model for provisional grades until the group-level criteria are met in a shadow-mode trial, where the model scores but humans grade. A model that fails the criteria should be used only for formative feedback, where the cost of error is lower and the student can act on it. If the vendor cannot provide disaggregated evidence, the provider should not accept “industry-leading” as a substitute.
3. The vendor proposes human review only for flagged students. Evaluate this workflow and propose an alternative that manages cost and fairness.
Read the worked answer
Reviewing only flagged students sounds efficient, but it concentrates human attention on the model's false positives and misses its false negatives. A student who is wrongly scored high will not be reviewed at all. In this case, the model under-scores EAL students on average, so the students most likely to be harmed are less likely to be flagged for review. The workflow therefore fails exactly where it is needed.
A better workflow uses risk-based sampling. Review all essays near grade boundaries, all essays where the model's confidence is low, and a random sample of the rest. Add a targeted sample of EAL and Māori and Pacific students, not because those students are suspect, but because the known bias means their scores need closer scrutiny. The random sample is essential: without it, the provider cannot estimate the model's error rate on the unflagged population.
Cost can be managed by tiering. The model can handle first-pass scoring and routine feedback. Human markers focus on boundaries, low-confidence cases, and a stratified moderation sample. If the budget cannot cover full review, the provider should reduce the scope of the pilot (e.g. 1,500 essays) rather than accept a workflow that disproportionately harms a group. A smaller, well-moderated pilot produces better evidence for the next semester.
There should also be an appeal route that does not require the student to prove bias. The provider should monitor appeal outcomes by group and publish the moderation results. The decision to use provisional grades is a governance decision, not just a procurement decision; the assessment committee, not only the vendor, should sign it off.