ACE-Expert cheat sheet

A test blueprint (content outline) specifies the domain structure, item counts per content area, and cognitive-level targets; every operational form is built to match it.

The passing standard, item pool size, and form length are set by policy before item writing begins; changing them mid-cycle invalidates prior equating.

Subject matter expert (SME) panels should be recruited against explicit, documented qualification criteria (experience, credentials, recency of practice).

Role separation: item writers, reviewers, and standard setters should not be the same individuals for the same item or panel to reduce bias.

A governance body (exam committee) holds authority for content validity decisions; psychometric staff hold authority for scoring and equating decisions.

Documented conflict-of-interest disclosure and recusal are required for every SME and committee member before item exposure.

Item and exam specifications are version-controlled; any post-publication change requires a documented change log and re-validation of affected items.

A defensible exam program maintains a technical manual documenting blueprint rationale, item development process, and psychometric procedures.

A well-written multiple-choice item has a single clearly correct key, plausible homogeneous distractors, and no cues (length, grammar, or 'all of the above') that reveal the answer.

Item difficulty (p-value) = proportion of examinees answering correctly; values near 0.0 or 1.0 provide little information.

Point-biserial (item-total) correlation indicates discrimination; negative or near-zero values flag a defective or miskeyed item.

Distractor analysis: each distractor should attract some examinees (typically >5%) and high performers should rarely select it.

Cognitive level should be assigned per item (e.g., recall, application, analysis) and matched to blueprint targets.

Avoid negative stems, double negatives, and unnecessary complexity; the stem should pose a clear, focused problem.

Items undergo content review (accuracy, relevance, fairness), editorial review, and sensitivity/bias review before pretesting.

New items are pretested (embedded or standalone) and only enter the operational pool after meeting difficulty and discrimination thresholds.

Classical test theory: observed score = true score + error; reliability = true-score variance / observed-score variance.

Cronbach's alpha and KR-20 estimate internal consistency; KR-20 applies to dichotomously scored items.

Standard error of measurement (SEM) = SD × sqrt(1 − reliability); used to build confidence bands around scores.

Item response theory (IRT) models item difficulty (b), discrimination (a), and guessing (c); the test information function summarizes precision across the ability range.

Equating places forms on a common scale; common-item (anchor) equating requires a stable set of linking items.

Scoring must handle missing responses and omitted items according to a documented rule (e.g., scored incorrect vs. not administered).

Differential item functioning (DIF) analysis detects items that behave differently across matched subgroups; flag for review, not automatic removal.

Reliability above ~0.90 is generally expected for high-stakes credentialing exams; lower values require justification or remediation.

The modified Angoff method asks panelists to estimate the probability that a minimally competent examinee answers each item correctly; item ratings are summed to a cut score.

The Bookmark method orders items by difficulty and panelists place a bookmark at the point a minimally competent examinee would master.

The contrasting-groups method derives a cut score from the score distributions of known-competent and known-incompetent groups.

Standard-setting panels require trained, representative SMEs and documented instructions; panelist judgment variability is a known limitation.

The cut score should be validated against external criteria (job performance, supervisor ratings, or criterion-group outcomes) where feasible.

The Hofstee method constrains the cut score using panelist-specified minimum/maximum acceptable pass rates and cut scores.

After equating, the cut score is held constant on the reporting scale so that the standard does not drift across forms.

Document the standard-setting method, panel composition, judgments, and rationale to support defensibility and future review.

Item exposure control limits how often an item is administered; overexposed items should be retired or rotated out.

A secure item bank requires access controls, audit logs, encryption, and role-based permissions for all users.

Test-center security includes candidate identity verification, proctor monitoring, and secure delivery environments.

Fairness review screens for construct-irrelevant content, cultural bias, and language that disadvantages subgroups unrelated to the construct.

DIF statistics (e.g., Mantel-Haenszel, IRT-based) supplement but do not replace expert fairness review.

Documented incident-response procedures cover suspected item compromise, security breaches, and score invalidation.

Reasonable accommodations (extended time, alternate formats) must preserve construct validity and be documented per candidate.

Regular security audits and item-harvesting detection (e.g., unusual answer patterns) are part of an ongoing security program.

The Kirkpatrick model evaluates training/programs at four levels: reaction, learning, behavior, and results.

Key performance indicators for an exam program include pass rate stability, item bank health, reliability, and candidate satisfaction.

Continuous improvement cycles (e.g., PDSA: Plan-Do-Study-Act) structure iterative review of exam processes and outcomes.

Item-bank health metrics include active item count, exposure rates, and the proportion of items flagged for review.

Psychometric trend monitoring tracks difficulty, discrimination, reliability, and pass rates across administrations to detect drift.

Stakeholder feedback (candidates, SMEs, employers) should be systematically collected and analyzed to inform improvements.

Regular technical reviews and re-validation cycles ensure the exam remains aligned with current practice and standards.

Root-cause analysis (e.g., 5 Whys, fishbone) is applied to recurring quality problems such as miskeyed items or scoring errors.