Scaling the External Oral
An external oral examination in Indonesian is marked by two assessors using a rubric with four criteria, each scored 1–5. Across the state, 1,840 students sat the oral. The raw total is out of 20. After marking, the examination authority notices that the mean raw score is 12.4 with a standard deviation of 3.1, while the target distribution for the subject has a mean of 13.0 and the same spread. A small group of schools in one region has a mean of 10.9, notably below the state mean, and their students are mostly from homes where Indonesian is not the first language. The authority must decide whether to apply a raw-to-scaled conversion, and whether the regional pattern indicates a marking problem or a cohort difference.
1. Explain what a raw-to-scaled conversion does and what it can and cannot fix in this situation.
Read the worked answer
A raw-to-scaled conversion maps raw totals onto a scaled distribution with a specified mean and spread, usually through a linear transformation or a percentile-based mapping. It can fix a mismatch between the raw marks and the intended reporting scale - here, lifting the mean from 12.4 to 13.0 while preserving the standard deviation of 3.1, so the rank order of students is unchanged. It can also make year-to-year results comparable if the paper difficulty varies. What it cannot do is fix validity problems. If the regional cohort scored lower because the oral tasks or topics disadvantaged students from non-Indonesian-speaking homes, scaling does not remove that disadvantage; it merely re-labels the scores. Scaling also cannot correct marker severity differences between assessors unless the conversion is built on marker-level statistics. So the authority should treat scaling as a reporting decision and treat the regional pattern as a separate question requiring evidence about task fairness and marker consistency.
2. The authority proposes to apply a bonus of +0.6 raw marks to all students in the low-scoring region to 'close the gap'. Evaluate this proposal.
Read the worked answer
This is the weakest option and should be rejected. A uniform regional bonus treats a group characteristic, not individual achievement, and would raise a strong student and a weak student in that region by the same amount, distorting the ranking both within the region and against the rest of the state. It also assumes the gap is caused by disadvantage rather than by teaching, cohort ability or marker severity, without evidence. If the gap is a marker-severity effect, the fix belongs in marker training and moderation, not in a student bonus. If the gap is a genuine cohort difference, a bonus is a form of group compensation that the assessment authority would need an explicit policy basis to apply, and it would be hard to defend to students elsewhere. A more defensible path is to investigate first: compare marker means within the region, check inter-rater reliability, review whether oral topics drew on culturally specific knowledge, and look at the cohort's performance on other components. Only then decide whether any adjustment is warranted, and if so, base it on evidence and apply it consistently.
3. Two assessors marked the same 60 recordings twice. Their scores differed by 2 or more points on 14 of the 60 recordings. What does this indicate and what should be done?
Read the worked answer
A discrepancy of 2 or more points on 14 of 60 recordings (about 23%) indicates poor inter-rater reliability on a 20-point scale with four criteria - roughly a 10% disagreement at that magnitude is a serious signal. It means a student's reported score depends substantially on who marked the recording, which undermines fairness and the defensibility of the external mark. The authority should first check whether disagreements cluster on particular criteria (for example, 'range' or 'interactional fluency' are often the vaguest), which suggests rubric descriptors need sharper anchors. It should then run moderation: a third, senior assessor re-marks a sample, and the group discusses the discrepant cases to calibrate. If discrepancies persist, the authority should consider double-marking all recordings from the affected assessors, or applying a marker-severity adjustment based on the moderation sample. Simply averaging the two discrepant scores is not sufficient, because averaging two unreliable judgements does not produce a reliable one. The rubric should be revised before the next examination cycle.