Assessment, Evaluation, and Learning Outcomes Measurement
This domain separates assessment of learning from evaluation of programmes. Candidates must know the levels, the methods, and the validity concerns.
Core concepts
- Assessment types: formative (during learning, feedback-oriented, low stakes) versus summative (end of learning, judgement-oriented, higher stakes). Diagnostic assessment precedes instruction.
- Kirkpatrick's four levels: reaction, learning, behaviour, results. Level 3 (behaviour) requires on-the-job observation or follow-up; level 4 requires organisational metrics. Most programmes stop at level 1 - a common weakness.
- Phillips ROI: adds a fifth level, monetary return, with isolation of training effects.
- Validity and reliability: a test is valid if it measures what it claims in the intended context; reliable if consistent. A reliable test can be invalid.
- Criterion-referenced versus norm-referenced: mastery against a standard versus comparison to a group.
- Item quality: avoid ambiguous stems, double-barrelled items, and cueing. Use item analysis (difficulty, discrimination).
- Evaluation designs: pre/post, control group, and time series; beware confounding and reaction bias.
Traps in questions
A frequent trap is treating a smile sheet (level 1) as evidence of learning. Another is confusing validity with reliability. Also watch for options that measure level 4 results without isolating training's contribution, or that use norm-referenced grading in a mastery programme.
Rule of thumb
Match the evaluation level to the claim being made. If the claim is "learners can do X", you need level 3 evidence, not satisfaction data. For test quality, ask: does the item measure the objective, and does it perform consistently across learners?
Worked example
A compliance programme reports 95% satisfaction. Leadership asks whether staff now follow the procedure. The correct answer is to collect observation or audit data on the procedure (level 3), not to cite the satisfaction score as proof.