Item Analysis in a Scholarship-Critical Test (Subject: Education)

Interpretation:

• Item 5 (p = .92, D = .10): very easy, almost no discrimination. For a norm-referenced ranking purpose it adds nothing; for a criterion-referenced mastery check it may be retained if the content is essential. The distractors are almost certainly non-functional - check whether any distractor drew more than a handful of responses.
• Item 12 (p = .48, D = .62): near-ideal. Moderate difficulty with strong discrimination. Retain unchanged.
• Item 27 (p = .21, D = -.18): the negative discrimination is the alarm. High scorers did worse than low scorers, which is the signature of a miskeyed answer, an ambiguous stem, or two defensible options. It is not simply a 'hard item' to be deleted.
• Item 33 (p = .65, D = .35): acceptable; retain, perhaps tighten one distractor.

Why Item 27 comes first: if the key is wrong, every total score in the file is wrong, and the two students who scored 34 and 35 have a legitimate grievance. Deleting the item changes all 120 totals; re-keying also changes all 120 totals but preserves the construct being measured. The sequence must be: inspect the item and the answer distribution, re-key or revise, re-score all scripts, then recompute item statistics and reliability, and only then decide on retention. Doing the deletion first destroys the evidence needed to judge the re-evaluation request fairly.

No. Quantify it:

SEM = SD × √(1 − r) = 5.2 × √(1 − .58) = 5.2 × 0.648 ≈ 3.37 marks.

A 95% confidence band around any student's obtained score is roughly ±1.96 × 3.37 ≈ ±6.6 marks. With scores spread over 40 marks, many students near the tenth rank are statistically indistinguishable from students ranked fifteenth or twentieth. Awarding a single high-value scholarship on the point estimate will misclassify real candidates.

What the institution should do:

1. Do not select on one test. Build a composite: this test plus a second parallel form or a structured interview/viva plus prior-semester results, with weights fixed in advance.
2. Use a band, not a line. Take all students within one SEM (about 3.4 marks) of the tenth rank into a shortlist and decide among them by a pre-declared tie-break - for example a 20-minute written task marked blind by two examiners.
3. Raise reliability if the test must carry the decision. By Spearman–Brown, reaching r = .80 with items of similar quality needs roughly 40 × [(.80 × .42)/(.58 × .20)] ≈ 115 items, which is impractical for a unit test - another argument for a composite.
4. Publish the rule before the next cycle so the selection is transparent and not seen as discretionary.

The proposal confuses spreading scores with measuring better.

Problems:

• Adding very hard items to compress the top lowers the mean and produces a floor pile-up; if the items are not in the blueprint, it also damages content validity - the test would no longer represent the unit taught, and the diagnostic purpose collapses.
• Deleting all weak items removes content coverage. A test that keeps only strongly discriminating items becomes a narrow measure of a few topics, which is poor for remediation even if it ranks students well.
• Two purposes, one instrument is the underlying flaw. Ranking for an award needs maximum discrimination and reliability; diagnosing weak areas needs broad, balanced content coverage including easy items on essential topics.

Defensible alternative:

1. Fix Item 27 (re-key or rewrite), then re-score and recompute.
2. Separate the purposes: a 40-item diagnostic test mapped to the content blueprint for remediation, and a composite selection procedure for the scholarship.
3. Retrofit the existing items rather than discard them: rewrite non-functional distractors on Item 5, tighten one distractor on Item 33, and pilot any new items on the next cohort before they carry weight.
4. Document the whole chain - item statistics, re-keying decision, re-scored totals, revised reliability - because the two re-evaluation requests make the file potentially contestable.