The 40-Item Test That Cannot Carry a Promotion Decision

The two indices must be read together; neither alone tells you what to do.

• Item 3 (p = .92, D = .08): far too easy and it separates nobody. Keep it only if it tests a curriculum-critical basic skill every Grade 8 student must have - then move it to the start as a confidence item. Otherwise revise or drop. It contributes almost nothing to the total score.
• Item 9 (p = .35, D = .55): hard but strongly discriminating. This is a good item - keep it. Verify only that it tests taught content and is not a reading-comprehension trap in disguise; if the language load is the difficulty, rewrite, do not delete.
• Item 14 (D = −.12): negative discrimination means better students got it wrong. Do not simply delete it. This is a diagnostic signal: check the scoring key, look for two defensible answers, ambiguous wording, or a misprinted diagram. Fix the cause, then re-check. Deleting it hides the defect.
• Item 22 (p = .78, D = .30): acceptable - moderately easy, still discriminating. Keep.
• Item 31 (p = .20, D = .05): very hard and non-discriminating. This usually means above-grade content, an unfamiliar context, or a multi-step item with an unrelated barrier. Check alignment with the taught curriculum, then revise or drop.

Aggregate: five flagged items is not fatal in a 40-item test, but combined with KR-20 = .54 it shows the test is measuring several different things weakly rather than one thing well.

Use the standard error of measurement.

SEM = SD × √(1 − r) = 4.1 × √(1 − 0.54) = 4.1 × √0.46 = 4.1 × 0.678 ≈ 2.8 marks.

A 95 per cent confidence band around an observed score is about ±1.96 × 2.8 ≈ ±5.5 marks.

Now take the student sitting exactly on the cut score: 16/40. The plausible range for that student's true score is roughly 10.5 to 21.5. The same band applies to a student on 18 and a student on 14. In practice, a wide group of students near 16 cannot be separated from each other by this instrument at all - the test is producing a coin-flip in exactly the region where the decision is hardest and most consequential.

Add three further problems: one item is probably mis-keyed (Item 14), two items are non-functional (3 and 31), and the reliability of .54 is low for a decision of this weight.

Conclusion: the test may inform promotion but cannot be the sole basis. Promotion should rest on the test plus continuous assessment, a portfolio of classwork, and - for borderline cases only - a short oral or a re-test on the specific competencies. If the score is retained as one input, borderline students (say 13–19) must be reviewed individually rather than cut mechanically.

Reject it as proposed, and explain why it would do harm.

Raising a cut score on a low-reliability, partly defective instrument does not restore standards; it increases false negatives - competent students failed for measurement error. It also changes the meaning of the test after the results are known, which is indefensible if any appeal is heard.

The real diagnosis is different. A mean of 21.4 (53.5 per cent) with a pass at 40 per cent means the class average is only just above the cut; the distribution, not the cut score, is the problem. The prior question is opportunity to learn: were the flagged items' topics actually taught, and for how long? A test of untaught content cannot set a standard.

What to do instead:

1. Fix the test first (key, ambiguity, alignment) and, if possible, re-administer or use a second form for borderline students.
2. Keep 40 per cent this term; a cut score must be set on content grounds - a panel judgement of what a minimally competent Grade 8 student should answer - not on a desired pass rate.
3. Establish the standard for next term, in advance, on a validated form, and publish the standard-setting method so it survives scrutiny.
4. Publish results with the measurement caveat, and record supplementary evidence for borderline cases.

Raising standards is done by improving instruction and the instrument, not by moving a line on a graph after the marking is finished.