The Angoff Panel That Disagreed With Itself
A behavioral-health licensure board convened a 12-member standard-setting panel using a modified Angoff method on a 160-item exam. Panelists rated items in three rounds. Round 3 mean minimum-passing percentage was 71% (SD 9.4). Two subgroups diverged sharply: the 4 panelists who were recent graduates rated items 8-12 points lower than the 8 experienced practitioners, and the divergence widened across rounds rather than converging. The board's prior cut score was 68%. The psychometrician computed the standard error of the cut score as ±3.1 points. The board chair wants to adopt 71% because it is close to the prior standard and avoids a politically difficult change.
1. Assess whether the panel's Round 3 result supports adopting 71%. What does the persistent subgroup divergence indicate?
Read the worked answer
Persistent, widening divergence across rounds is a warning sign, not convergence. Modified Angoff assumes panelists converge on a shared conception of the minimally competent candidate. Divergence that grows suggests the two subgroups are anchoring on different referents - recent graduates may be imagining a candidate like themselves, experienced practitioners a candidate they would supervise.
Implications:
- The 71% mean masks a bimodal distribution. A mean with SD 9.4 and a structured 8-12 point split is not a stable consensus estimate; the standard error of ±3.1 likely understates true uncertainty because it assumes independent, homogeneous judgments.
- Proximity to the prior 68% is not evidence of validity. The chair's rationale is a political convenience dressed as stability. A cut score is defensible because the method and panel were sound, not because it matches last year.
- The 3-point gap between 68% and 71% is within the ±3.1 SE, so the data cannot distinguish the two. Adopting 71% as if it were a precise estimate misrepresents the evidence.
Recommendation: do not adopt 71% on this basis. Either reconvene with a facilitated discussion to surface and reconcile the referent definitions, or report a confidence band and select within it on policy grounds with explicit justification.
2. The board insists on a decision before the next administration in six weeks. Design a defensible path forward and state what you would document.
Read the worked answer
A defensible six-week path:
- Diagnose the divergence. Hold a structured 90-minute calibration session with both subgroups. Ask each to describe the minimally competent candidate in behavioral terms and rate 5 anchor items live. If the gap persists after shared referents, treat it as genuine judgment variance, not a facilitation failure.
- Weight or report both. If divergence persists, either (a) retain the full panel mean but widen the reported uncertainty to reflect the subgroup structure, or (b) present the board with the range (approximately 66-76%) and require a policy decision, documented with rationale.
- Cross-check with consequential evidence. Compare candidate pass rates at 68%, 71%, and the band edges against known competence indicators (supervisor ratings, first-year disciplinary events) if available.
- Decide and document. Record: the method, panel composition and its limitation, round-by-round data, the divergence and how it was handled, the SE and its caveats, the board's chosen cut score, and the explicit rationale. The documentation must let a future auditor see that the board chose with eyes open, not that it hit a convenient number.
If the board cannot meet, default to the prior 68% for one cycle with a written commitment to reconvene - changing a cut score on unstable data is worse than deferring.