Interpreting a Screening Test for a Scholarship Shortlist
A scholarship program screens 2,000 applicants with a 100-item aptitude test. The top 200 scorers are shortlisted for interview. A reviewer notices that a candidate who answered 78 items correctly was shortlisted, while a candidate with 76 correct was not, even though the second candidate left 20 items blank and the first left 2 blank. The test is scored as one point per correct answer with no penalty for blanks. The program director wants to know whether the shortlist is fair, and whether a correction is possible without rescoring everyone by hand. Historical data show the test has a reliability of 0.85 and that guessing on unanswered items would yield about 25 percent correct.
1. Explain how the blank-answering strategy affects the ranking and whether the two candidates were treated inconsistently.
Read the worked answer
Under one-point-per-correct with no penalty, a blank and a wrong answer score identically - zero. So the candidate with 76 correct and 20 blanks scored 76; the candidate with 78 correct and 2 blanks scored 78. The shortlist cut fell between them, so the ranking is internally consistent with the stated rule: 78 > 76.
The fairness concern is different. The two candidates faced the same rule, but the rule rewards answering volume only when guesses are correct. If the 78-correct candidate guessed on many items and got some right, the score partly reflects guessing luck. The 76-correct candidate with 20 blanks may have known more but declined to guess. The rule is consistent but not necessarily valid as a measure of ability.
A correction is possible: since blanks can be treated as attempted-but-unknown, one can estimate an ability-adjusted score, e.g., correct minus a fraction of wrong answers, or treat blanks as 25 percent correct in expectation. Applying 0.25 × 20 = 5 expected points to the second candidate's blanks gives an expected score of about 81, which would move them above 78. This shows the ranking is sensitive to the scoring convention.
2. The director asks whether reliability 0.85 is high enough to justify a hard cutoff at rank 200. Advise with reasoning.
Read the worked answer
Reliability 0.85 means about 15 percent of score variance is error. Near a cutoff, small error translates into many misclassifications. The standard error of measurement is SD × √(1 − r). Without the SD we cannot give an exact number, but the logic holds: candidates clustered near rank 200 have scores within a few points of each other, and a few points is well within measurement error for a test of this reliability.
Advice: do not treat rank 200 as a sharp line. Either (a) shortlist a band, e.g., ranks 180–220, and interview all of them, or (b) use the test only to eliminate clearly weak candidates and let interview or a second measure decide near the boundary. A hard cutoff is defensible only if the program accepts that some equally able candidates will be excluded by measurement noise. Document the band rule in advance so it is not seen as favoritism.
3. A candidate claims the 20-blank strategy was taught by a coaching center. Does this change the fairness analysis?
Read the worked answer
It changes the interpretation but not the arithmetic. If coaching centers teach 'leave blanks when unsure,' then blank-leaving becomes a strategy correlated with access to coaching, not with ability. Candidates without coaching may guess, gaining expected points (0.25 per blank) that coached candidates forgo. Under the current rule, guessing is weakly advantageous in expectation, so the coached blank-leavers are penalized relative to guessers of equal ability.
Fairness implication: the scoring rule interacts with test-taking strategy, and strategy access is unevenly distributed. The remedy is a rule that makes strategy irrelevant - for example, instruct all candidates to answer every item (guessing is then uniform) or apply a correction for guessing to everyone. The claim itself does not prove unfairness, but it identifies a mechanism by which the rule disadvantages some groups. The program should verify whether coaching access differs across regions or income groups before deciding how much weight to give the claim.