ACE-Expert common mistakes
ACE-Expert · Understand the traps and how to avoid them.
Treating a criterion-referenced certification cut score as if it were a norm-referenced percentile, and justifying the passing standard by noting that 'about 70% of candidates pass.'
Candidates carry over intuition from academic grading and norm-referenced testing, where a pass rate is a meaningful target, and conflate the standard with the resulting pass rate.
What to do instead
State explicitly that the cut score is derived from the knowledge/skill required for competent practice (content-referenced), and that the pass rate is an outcome to be monitored, not a criterion for setting or adjusting the standard.
When asked to evaluate an item, judging it solely on p-value and declaring a low p-value item 'bad' without considering whether the low performance reflects a real difficulty in the construct or an item flaw.
Difficulty indices are the most familiar and easily computed statistic, so candidates anchor on them and skip the discrimination and distractor analysis that distinguish 'hard but valid' from 'flawed.'
What to do instead
Evaluate items on a coordinated set of evidence: difficulty, discrimination (point-biserial or equivalent), distractor functioning, and content alignment; only flag an item when multiple indicators plus content review converge.
Writing a stem that can be answered by test-wise guessing (e.g., the longest option is correct, or grammatical cues from the stem eliminate distractors) and submitting it as complete.
Item writers focus on content accuracy and forget that item-writing flaws are a separate quality dimension; they also lack a self-review checklist habit.
What to do instead
Apply a structured item-writing flaw checklist (length cueing, grammatical consistency, 'all of the above,' absolute terms, negatively worded stems) before submission, and have a second reviewer apply the same checklist independently.
Describing a standard-setting method by name (e.g., Angoff, Bookmark) without specifying the operational procedure, panel composition, or how judgments are translated into a cut score.
Candidates memorize method labels as vocabulary rather than as procedures, so they can name the method but cannot defend the judgmental process or its documentation.
What to do instead
For each method, be able to describe the panel task step by step, the training and feedback rounds, the aggregation rule, and the evidence used to validate the resulting standard.
Responding to a test-security incident by immediately invalidating scores or removing the item, without first documenting the incident, preserving evidence, and determining scope.
Candidates want to demonstrate decisive action and skip the investigative and documentation steps that make a response defensible.
What to do instead
Follow the incident-response sequence: contain and preserve evidence, document facts and timeline, assess scope and exposure, consult policy and stakeholders, then decide on remedies with a written rationale.
Confusing reliability with validity, e.g., arguing that a test is valid because its internal consistency coefficient is high.
Both are reported as single coefficients, and reliability is easier to compute and explain, so candidates substitute the familiar statistic for the harder validity argument.
What to do instead
Frame validity as an argument built from multiple evidence sources (content, response process, internal structure, relations to other variables, consequences) and treat reliability as a necessary but insufficient precondition.
Designing a fairness review that only checks for obviously offensive language, without examining differential item functioning, construct-irrelevant difficulty, or accessibility for candidates with disabilities.
Candidates equate fairness with sensitivity review, which is the most visible and best-known component, and overlook the psychometric and accessibility dimensions.
What to do instead
Build fairness review as a multi-stage process: sensitivity and bias review at authoring, statistical DIF analysis after administration, review of construct-irrelevant barriers, and documented accommodations procedures.
Proposing to 'improve the program' by adding more items or raising the cut score, without linking the change to a measured problem or an evaluation question.
Candidates treat program evaluation as a list of generic improvements rather than a systematic inquiry tied to data and stakeholder needs.
What to do instead
Anchor each recommendation in an evaluation question, a data source, and a success indicator; describe the evaluation model (e.g., input-process-outcome) and how findings will feed a documented improvement cycle.
Describing a governance structure only as an org chart, without specifying decision rights, accountability, conflict-of-interest controls, or how policy changes are approved and versioned.
Candidates summarize who reports to whom and stop there, missing the procedural controls that make governance auditable.
What to do instead
For each governance body, state its charter, decision authority, membership and COI rules, meeting and documentation requirements, and the change-control process for policies and forms.
Treating a passing standard or item bank as permanent, and omitting any plan for periodic review, equating across forms, or revalidation after content or job changes.
Candidates view psychometric work as a one-time event rather than a maintenance lifecycle, so they describe the initial study but not the sustainment activities.
What to do instead
Include a maintenance plan: scheduled standard review, equating and form-assembly procedures, item exposure and bank-health monitoring, and triggers for revalidation (job analysis updates, major content changes).