Scores Up, Competence Flat
A financial services firm launched a 6-week online course on anti-money-laundering detection for 300 analysts. Completion is 94 percent and average end-of-course quiz scores rose from 71 to 84 percent between the first and second cohorts. The compliance director is pleased and wants to declare the program a success. However, the quality team notices three things: the quiz is the same 20 items each cohort, the second cohort had two extra hints added to the platform, and case-review data shows that suspicious-activity report quality has not moved since launch. The learning team's dashboard currently reports only completion and quiz average.
1. Evaluate whether the reported improvement is credible evidence of learning, and identify the measurement flaws.
Read the worked answer
The improvement is not credible as evidence of learning. Three flaws compound. First, the same 20 items were reused across cohorts, so the second cohort could have benefited from item exposure - earlier learners talk, and the items circulate. Second, two hints were added to the platform between cohorts, which lowers the difficulty of the very task being measured; a score gain under easier conditions is not a gain in competence. Third, the metric is a single post-test with no comparison group and no baseline of ability, so regression to the mean and selection differences between cohorts are uncontrolled.
The dashboard compounds the problem by reporting only completion and average score - both activity and immediate-reaction measures. Neither tells the compliance director whether analysts can now spot and report suspicious activity. The honest conclusion is that the program is well attended and the quiz is being passed; whether learning occurred is unmeasured.
2. Redesign the measurement approach, naming the levels you would measure and one concrete metric for each.
Read the worked answer
Move to a four-level design. At reaction, measure perceived relevance and confidence, not just satisfaction - a two-item scale asking whether analysts can apply the content to their own caseload. At learning, use a pre/post design with a refreshed item bank each cohort and a scenario-based assessment scored against a rubric, so scores reflect judgment rather than item familiarity. At behavior, sample case files at 30 and 90 days and rate SAR quality against a rubric covering completeness, accuracy of the suspicion narrative, and timeliness; this is the metric that should have been on the dashboard from day one. At results, track downstream indicators such as the proportion of SARs accepted without rework and the false-positive rate on alerts.
Also change the dashboard so it shows the behavior and results measures alongside completion, and report a baseline before claiming improvement. The point is to make the claim falsifiable: if SAR quality does not move at 90 days, the program should be revised, not celebrated.