Setting a Passing Standard for a New Competency-Based Exam
A regulatory body is launching a new exit examination aligned to a revised competency framework. The exam has 150 single-best-answer items and a practical station. The panel of 12 subject-matter experts includes 4 clinicians, 4 academics, and 4 recent graduates. A prior exam used a norm-referenced rule (pass the top 70%). Stakeholders want a defensible, criterion-referenced standard. The panel is split: some want a fixed 60% cut, others want a borderline-group method. Time allows one standard-setting workshop of two days. Historical first-attempt pass rates under the old rule were 71-78%.
1. Recommend a standard-setting method and justify it against the alternatives.
Read the worked answer
Recommendation: a modified Angoff (or Yes/No) method for the written paper, with a documented borderline-group exercise as triangulation, and a separate checklist-based standard for the practical station.
Why not a fixed 60%: A fixed percentage is arbitrary; it is not tied to the minimum competent performance and cannot respond to item difficulty. It is easy to defend administratively but weak psychometrically.
Why not the old norm-referenced rule: Passing the top 70% means the cut moves with cohort ability; a weak cohort could pass candidates who are not competent, and a strong cohort could fail competent ones. For licensure, the standard must be absolute, not relative.
Why modified Angoff: It directly operationalizes 'minimally competent' by having experts estimate the probability that a borderline candidate answers each item correctly. It is feasible in two days for 150 items with 12 panelists, produces a documented cut score, and is widely accepted for defensibility. The borderline-group method adds a reality check but requires candidate data, so use it after the first administration to validate the Angoff cut.
2. The Angoff-derived cut is 58%, but the panel notes that recent graduates rated many items much lower than clinicians did. How should this disagreement be handled?
Read the worked answer
Handle it as a data-quality signal, not by averaging it away.
- Inspect the pattern. If recent graduates rated items lower systematically, they may be judging from their own recent performance rather than from the minimum standard - a known bias in Angoff panels. Clinicians may overestimate what a borderline candidate knows.
- Structured discussion round. Run a second rating round with item-level discussion, focusing on items with the largest discrepancies. Require panelists to justify ratings against the competency definition.
- Weighting: Do not weight panelists by role arbitrarily. Instead, use the consensus after discussion and record the rationale for each large shift.
- Sensitivity analysis: Report the cut under each subgroup's ratings. If the range is narrow (e.g., 56-60%), the standard is robust. If wide, flag the exam as provisional and plan to validate against first-year practice data.
Key point: Disagreement is expected and should be resolved through evidence and discussion, not by discarding a group's input or by splitting the difference mechanically.
3. How should the practical station's standard be set, and how should the two components combine?
Read the worked answer
Practical station: Use a checklist or behaviorally anchored rating scale with a defined minimum number of critical behaviors. Critical items (e.g., aseptic technique, patient identification) should be mandatory - failing a critical item fails the station regardless of total score. Set the threshold by expert consensus on the minimum acceptable performance, not by a percentage of the maximum.
Combining components:
- Decide whether the exam is compensatory (high written score can offset a weak practical) or conjunctive (must pass both). For licensure, a conjunctive rule is usually safer because a candidate who cannot perform safely should not pass on written strength alone.
- If compensatory, weight components by importance and set a combined cut; document the weighting rationale.
- Report each component separately so remediation can be targeted.
Defensibility: Record the method, panel composition, and decision rules in a technical report before results are released.