A hiring panel, a chair's email, and five post-discussion ratings
At a mid-sized hospital, a committee of five evaluates eight finalists for a nurse-manager post. Each member independently completes a structured interview sheet with behaviorally anchored ratings plus a single "overall fit" item. Before the day-long session, the chair emails the group a summary stating that candidate C is "clearly the strongest on paper." After discussion, four of five members rate C highest; the lone dissenter, who has the most supervisory experience of anyone on the panel, rates candidate F highest. The pre-discussion sheets show that the chair's own ratings of C were 1.5 SD above the panel mean for that candidate, and that across all candidates each rater's "overall fit" rating correlated r = .78 with their rating of "likability in the interview," compared with r = .31 with a work-sample score. The chair proposes averaging all post-discussion ratings and extending an offer to C.
1. Which social-psychological processes could generate this pattern, and which of them do the described data actually support?
Read the worked answer
Candidate processes: informational and normative conformity (the chair's pre-meeting email supplies an anchor and a perceived consensus); anchoring on the chair's stated ranking; halo effect, in which global likability bleeds into ratings of distinct competencies; group polarization, in which discussion moves the group toward a more extreme version of its initial lean; shared-information bias, in which the panel discusses what everyone already knows rather than each member's unique evidence; and similarity or affinity bias toward a candidate who resembles the majority.
What the data support. The halo interpretation is directly supported: r = .78 between overall fit and interview likability, against r = .31 with a work-sample score, is a clear signature of a global impression driving specific judgments, and it also means the fit item is largely a likability item. The chair's 1.5 SD elevation is a leniency/anchoring signal for that rater, not proof that the panel conformed. Conformity and polarization are plausible but inferred - we see the pre-meeting email and the post-discussion unanimity, not the intervening conversation.
The dissent matters. A single holdout with the most relevant expertise is exactly the information a conformity explanation predicts would be suppressed; treating it as noise discards the panel's best-evidenced judgment.
2. The chair wants to average post-discussion ratings. How should pre- and post-discussion ratings be used, and what decision rule is defensible?
Read the worked answer
Post-discussion ratings are contaminated as independent evidence. Averaging them treats consensus as accuracy, when consensus can be manufactured by the chair's email, by normative pressure, and by polarization. The average will also hide the dissent that matters.
A defensible procedure:
- Weight the pre-discussion, independently completed ratings as the primary index. They were collected before social influence, so they are more valid measurements of each rater's evidence.
- Restructure the discussion: before any open deliberation, have each member state the two strongest pieces of evidence for their top and second choice; explicitly invite disagreement and require the dissenter to be heard first. Then have members re-rate independently rather than converge aloud.
- Drop or re-specify the "overall fit" item, which appears to be a likability measure; weight behaviorally anchored, job-relevant dimensions and the work sample more heavily.
- Mechanically combine scores using a rule fixed in advance, rather than discussing to consensus. Actuarial combination generally outperforms clinical aggregation of impressions.
On the merits, candidate C is not established as the strongest - the panel has one high-anchored chair, a likability-driven fit item, and a well-qualified dissenter. The offer decision should wait for re-rating.
3. What psychometric problems arise from aggregating five raters across eight candidates, and how should the committee describe the reliability of its process?
Read the worked answer
The design is too small to estimate reliability credibly. With eight targets and five raters, an intraclass correlation will have very wide confidence intervals; a value of, say, .60 could be consistent with anything from poor to excellent. Reporting a point estimate without intervals, or treating it as a stable property of the interview, overstates what the data can show.
Specific issues:
- Restriction of range. Eight finalists are a pre-selected, high-ability sample; restricted variance depresses correlations and attenuates reliability estimates, so a modest ICC may understate the instrument's quality.
- Single-item measurement. The "overall fit" item has unknown reliability and is confounded with likability; reliability should be estimated on multi-item dimension scores.
- Rater effects. The chair's 1.5 SD leniency is a systematic rater effect, not random error. A multi-facet or generalizability-theory framing - separating candidate, rater, and item facets - is the appropriate model rather than a single ICC.
- Common method variance. All ratings come from the same interview, so likability and fit share method and occasion; the .78 is partly method.
Practical description: report the process's limits honestly and rely on structured, pre-specified, mechanically combined evidence, not a reliability coefficient computed on eight people.