The Item That Everyone Flags but Nobody Can Fix
A governance committee reviews item-level fairness flags monthly. One item on a professional-ethics exam has been flagged by three separate review processes: a candidate comment alleging cultural bias, a DIF analysis showing a C-level difference for one subgroup, and a content reviewer noting the scenario assumes a legal framework common in one country. The item has been in the pool for six years, has good discrimination (point-biserial 0.41), and is used on every form. The item writer, a senior subject-matter expert, disputes all three flags and argues the item tests a universal ethical principle. The committee has 30 minutes and a quorum of five.
1. How should the committee weigh three different kinds of flag against one another, and what does the item's strong discrimination tell you?
Read the worked answer
The three flags are not equivalent and should not be averaged:
- Candidate comment is weak evidence on its own - it is an allegation, not a finding - but it is a signal that prompts investigation.
- DIF is statistical evidence of differential performance after matching on ability. It does not prove bias, but it shifts the burden to the item writer to explain the difference on construct-irrelevant grounds.
- Content review finding a jurisdiction-specific legal assumption is the strongest flag here, because it identifies a plausible construct-irrelevant cause that explains the DIF. The three flags converge on one mechanism.
Strong discrimination (0.41) is not exculpatory. A biased item can discriminate well precisely because the bias gives one subgroup an advantage that correlates with ability. High discrimination tells you the item is measuring something reliably; it does not tell you it is measuring the right thing.
The writer's "universal principle" defense is a claim, not evidence. The committee should ask: does the item's correct answer depend on the jurisdiction-specific framework, or only its scenario dressing? That is testable by rewriting the stem with a neutral framework and re-reviewing.
2. The committee has 30 minutes and five members. What decision process and outcome would you recommend, and what would you document?
Read the worked answer
Process (30 minutes):
- 5 min - confirm the three flags and their evidence quality; note the DIF effect size, not just its presence.
- 10 min - ask the writer one focused question: identify the specific stem language that encodes the jurisdiction-specific framework. If the writer can point to it, the fix is editorial; if not, the content flag stands.
- 10 min - decide disposition. Given converging flags and an identifiable mechanism, the defensible interim action is to remove the item from live forms pending revision, not to keep it while debating. A six-year-old item on every form is also an exposure risk.
- 5 min - assign the revision and re-review date.
Outcome: suspend the item; direct the writer to produce a jurisdiction-neutral revision; re-run DIF on the revision before reinstatement.
Document: the three flags and their evidence, the writer's response, the specific mechanism identified, the suspension decision and rationale, the revision assignment with owner and date, and the reinstatement criteria. The record must show the committee acted on converging evidence rather than deferring to seniority.