Scoring, Standard Setting, and Psychometrics
This domain links how scores are computed to how the passing standard is set and how items behave statistically. Start with scoring: raw scores are converted to scaled scores for comparability across forms; equating places scores from different forms on a common scale so that a given scaled score means the same thing regardless of form difficulty. Know the difference between norm-referenced (relative to a group) and criterion-referenced (relative to a standard) interpretation.
Standard setting is the formal process of defining the passing score. The modified Angoff method has SMEs estimate the probability that a minimally competent candidate would answer each item correctly; the Bookmark method has panelists place a cut score on an ordered difficulty scale. Both produce a cut score justified by expert judgment, not by a fixed pass rate. A fixed pass rate (e.g., "top 70% pass") is a norm-referenced shortcut and is generally inappropriate for certification.
Psychometrics: item difficulty (p-value) is the proportion answering correctly; item discrimination describes how well an item separates high and low performers; reliability concerns consistency of scores; validity concerns whether scores support the intended interpretation. Differential item functioning (DIF) flags items that behave differently for comparable candidates from different groups.
Traps: (1) Equating a scaled score with a raw score - they differ. (2) Confusing reliability (consistency) with validity (meaning). (3) Assuming a high p-value is always good - very easy items may not discriminate. (4) Thinking the cut score equals the average score of a panel.
Rule of thumb: validity is the umbrella; reliability is necessary but not sufficient for it. Worked example: an item answered correctly by 92% of candidates and by both high and low performers equally has a high p-value but poor discrimination - a candidate for revision.