Item Writing Workshop Dilemma
A certification board is developing a new exam. They hold a two-day item-writing workshop with 12 subject matter experts (SMEs). The SMEs write 200 new multiple-choice items. The exam blueprint requires 120 items. After the workshop, the items are reviewed by a psychometrician who finds: 30 items are flawed (ambiguous, multiple correct answers, or trivial), 20 items are too easy (p-value > 0.95 based on SME judgment), 15 items are too hard (p-value < 0.20), and 10 items are miscoded to the wrong domain. The remaining 125 items are acceptable. The board wants to administer the exam in three months. The SMEs are volunteers and have limited availability for additional work. The budget allows for one more half-day review session with the SMEs.
1. What should be the priority for the half-day review session, and how should the board decide which items to keep?
Read the worked answer
The priority should be to address the 30 flawed items first, as they are the most problematic for validity. The 10 miscoded items can be recoded by the psychometrician without SME input if the domain definitions are clear, but SME confirmation is ideal. The 20 too-easy and 15 too-hard items are not necessarily invalid; they may be kept if the exam needs a range of difficulty, but for a criterion-referenced exam, items near the cut score are most valuable. The board needs 120 items. Currently, 125 are acceptable, which is just enough, but they may want a buffer for future forms. The half-day session should focus on: (1) reviewing the 30 flawed items to either fix or discard them, (2) confirming recoding of the 10 miscoded items, and (3) evaluating the too-easy and too-hard items to see if they can be improved or should be replaced. The board should keep the 125 acceptable items as a base, then decide how many of the flawed/easy/hard items to salvage. They need at least 120, so they can afford to discard all 30 flawed and still have 125, but that leaves no buffer. They should aim to salvage at least 10-15 items from the flawed/easy/hard pool to have a buffer. The SMEs should be given the items in advance and asked to focus on the most critical flaws.
2. How should the board handle the SMEs' limited availability and the need for a defensible item review process?
Read the worked answer
The board should provide clear instructions and materials to SMEs before the session, including the item-writing guidelines, the blueprint, and the specific issues with each flagged item. During the half-day, they should use a structured review process: for each flawed item, SMEs discuss and vote on whether to revise or discard. For the too-easy/hard items, they can quickly rate whether the item is still relevant and fair. The psychometrician can facilitate and document decisions. To ensure defensibility, the board should keep records of the review, including SME qualifications, the criteria used, and the rationale for keeping or discarding items. If SMEs cannot reach consensus, the item should be discarded. The board might also consider having a smaller subgroup of SMEs do a follow-up review via email for items not resolved. Given the time constraint, the board should prioritize items that are closest to the cut score or that cover critical domains. The goal is to have a sufficient number of valid items for the exam, even if it means using some less-than-ideal items with documented limitations.