Standard Setting Method Choice
A new credentialing exam is being developed. The exam has 180 items, covering 6 domains. The passing standard must be set. The board is considering three methods: modified Angoff, Bookmark, and contrasting groups. The exam is computer-based and will be linearly administered. The item pool is new and has not been pretested. The SMEs are available for a two-day standard-setting meeting. The board wants a defensible, legally robust standard. The exam is intended to certify entry-level competence. There are 500 candidates in the first administration. The board has a limited budget for psychometric support.
1. Which standard-setting method is most appropriate given the constraints, and why?
Read the worked answer
The modified Angoff method is most appropriate. It is widely accepted, legally defensible, and feasible with a two-day meeting. It requires SMEs to estimate the probability that a minimally competent candidate would answer each item correctly. It works well with new item pools because it does not require examinee response data, unlike Bookmark or contrasting groups. Bookmark requires an ordered item booklet based on item difficulty, which is not available for new items without pretesting. Contrasting groups requires data from known groups (competent vs. not), which is not available for a new exam. Modified Angoff is also cost-effective as it can be completed in a two-day meeting with SMEs. It provides a clear cut score and documentation. However, because items are new, SMEs may find it challenging, but training and discussion can mitigate this. The board should ensure SMEs are properly trained and that the process includes multiple rounds and feedback.
2. What steps should the board take to ensure the standard-setting process is defensible and legally robust?
Read the worked answer
The board should: (1) select SMEs who are representative of the profession and trained in the method; (2) provide clear definitions of minimally competent candidates; (3) use a structured process with multiple rounds (e.g., three rounds) and facilitate discussion; (4) collect and document SME judgments and rationale; (5) compute the cut score as the average of SME ratings, and consider adjusting for item difficulty if data becomes available; (6) conduct a sensitivity analysis (e.g., compare with other methods if possible); (7) have the board approve the cut score and document the rationale; (8) ensure the process is transparent and consistent with professional standards (e.g., AERA/APA/NCME). Legal robustness also requires evidence of validity, including the relationship between cut score and job performance, but that may come later. The board should also consider the consequences of the cut score (e.g., pass rate) and adjust if it seems unreasonable, but not solely based on pass rate. Finally, maintain records for potential legal challenges.