Structured Interview Panel Scoring Drift
You are an HR specialist running a structured interview panel of three raters for a supervisory position. The panel uses a behaviorally anchored rating scale with five competencies scored 1-5. After the first two candidates, you notice that Rater A has given scores of 5 on every competency to both candidates, while Rater B has given mostly 2s and 3s. Rater A mentions that one candidate 'reminds me of myself at that stage.' A third candidate is scheduled in 30 minutes. The hiring manager wants all scores finalized today to meet the certificate issuance deadline.
1. What problem is present in the panel data, and what should you do before the third interview?
Read the worked answer
The pattern indicates likely rater leniency/severity and halo effects: Rater A is clustering at the ceiling and appears to be rating on similarity ('reminds me of myself') rather than on the anchors. Rater B is clustering low. Neither pattern is evidence that the candidates differ; it is evidence the raters are applying the scale inconsistently. Before the third interview, pause and conduct a brief calibration: review the behavioral anchors together, confirm what a 1, 3, and 5 look like with concrete examples, and remind raters to score each competency independently based on observed evidence. Do not simply average the divergent scores as if they were comparable. The judgment is recognizing that a structured process can still produce invalid data when raters drift, and that calibration must happen before more data is collected.
2. The hiring manager says to just average the scores and move on. How do you handle this while protecting the integrity of the process?
Read the worked answer
Explain that averaging lenient and severe scores does not produce a valid ranking; it can mask that the raters were measuring different things. Recommend a documented calibration discussion in which raters justify scores against the anchors and revise only where the evidence supports it, then finalize. If the manager still insists on a raw average, document the concern and the calibration steps taken, and note the rater-drift issue in the panel record. If your agency has a delegated examining or merit system authority, consult them. The judgment is balancing a deadline against the requirement that selection be based on job-related, consistently applied criteria, and ensuring the record shows what was done and why.