The Supervisor's Rating
A company uses a supervisor rating scale to decide promotions. Ratings are on a 1-5 scale, and 80% of employees receive a 4 or 5, so the scale barely discriminates. The company also gives a 40-item 'aptitude test' with a published reliability of 0.91, but the local HR team administers it with a 15-minute limit although the manual specifies 35 minutes. Last year the test correlated 0.18 with later supervisor ratings. A candidate who scored in the top 5% was promoted and then performed poorly; a candidate in the bottom 30% became the branch's best performer. HR now proposes to drop the test and promote on interview impression alone.
1. Diagnose the measurement problems in this system, separating problems with the criterion, the predictor, and the administration.
Read the worked answer
Criterion problem (the supervisor ratings). With 80% of ratings at 4 or 5, the scale has severe restriction of range and a ceiling effect. A criterion that does not discriminate cannot correlate strongly with anything, so the observed 0.18 is partly an artefact of a crippled criterion, not proof that the test is useless. Ratings may also carry halo error and leniency bias. Fix: force distribution or behaviourally anchored rating scales with defined anchors, plus objective indicators (sales, error rates, absenteeism) where available.
Predictor problem (the test itself). The published reliability of 0.91 applies to the test as standardised. Reliability is a property of scores under specified conditions, not of the booklet. A 15-minute administration of a 35-minute test changes the construct being measured - it becomes a speed test, not the intended aptitude measure - and truncates the ceiling, so high-ability candidates are penalised. Local reliability must be re-estimated, not assumed.
Validity problem. A correlation of 0.18 with a restricted criterion is uninformative. Before discarding the test, re-run the analysis with a corrected criterion and correct timing. Also compute incremental validity: does the test add anything over interview and experience?
The two anecdotes. One top-5% failure and one bottom-30% success are exactly what a probabilistic predictor produces. A valid test raises the base rate of good decisions; it does not guarantee any individual outcome. Reasoning from two cases is the availability heuristic applied to selection.
HR's proposal. Promoting on interview impression alone replaces a weak but measurable predictor with an unstandardised one prone to similarity bias, first-impression effects, and low inter-rater agreement. That is a downgrade, not a fix.
2. Recommend a decision rule the company could adopt, and explain how you would justify it to a manager who says 'tests don't work here.'
Read the worked answer
Recommended rule. Use a multiple-hurdles approach rather than a single cut-off:
- Screen on the corrected aptitude test with a modest cut (for example, the lower quartile) to remove clear mismatches, not to rank finely.
- Structured interview with the same scored questions for all candidates and at least two trained raters, to raise inter-rater reliability.
- Work sample or probationary task where feasible, which typically predicts job performance better than impressionistic interviews.
- Combine scores with explicit weights (for example, test 30%, structured interview 40%, work sample 30%) decided in advance, not after seeing candidates.
- Review the rule annually against the improved criterion, and monitor for adverse impact across groups.
Justifying it to the sceptical manager.
- Concede the point: the current test, as administered, is not working. The 15-minute limit is a deviation from the manual and turns it into a different test. Fix the administration before judging the instrument.
- Explain that the 0.18 figure is depressed by a criterion on which 80% of people score the same. You cannot detect a real relationship when the outcome barely varies.
- Use the two anecdotes directly: a probabilistic predictor will always produce exceptions; judging a test by its most memorable case is a reasoning error, not evidence.
- Point out that 'interview impression alone' has its own error rate, is harder to audit, and is more vulnerable to bias - so it is not the safe default.
- Offer a pilot: run the corrected test and structured interview for one cycle, then compare against actual performance. A time-limited trial with pre-specified criteria is more persuasive than an argument.