Scores Up, Performance Flat
A software company rolled out a new customer-support workflow. Level 1 evaluation (reaction) averaged 4.4/5. Level 2 (a 30-item knowledge test) showed a mean of 86% with a 92% pass rate, up from 71% on the pre-rollout baseline. Six weeks later, the support director reports that average handle time is unchanged and escalation rate has risen 8%. The L&D manager wants to declare the training a success based on the Level 2 results and close the project. The knowledge test was written by the training vendor and closely mirrors the e-learning screens. No observation of live calls has been done. The workflow tool logs show that 40% of agents use a workaround that bypasses the new triage step.
1. Evaluate the evidence. What can and cannot be concluded, and what is the most probable explanation for the pattern?
Read the worked answer
What can be concluded: learners can pass a test that mirrors the e-learning. That is a test-format artifact, not evidence of transfer. A 30-item test written alongside the course and resembling its screens measures recognition of course content, which is why it rose from 71% to 86% while behavior did not move.
What cannot be concluded: that the workflow training worked, or that it failed. Level 1 and Level 2 data are silent on transfer.
The most probable explanation is a workaround-driven transfer gap. The tool logs are the strongest signal in the dataset: 40% of agents bypass the triage step, which would explain unchanged handle time (the workaround preserves the old flow) and a rising escalation rate (triage is skipped, so cases reach the wrong queue). The likely causes are environmental, not motivational: the tool may be slow, the triage step may conflict with an incentive or a legacy habit, or agents may not have had practice on realistic cases.
Declaring success and closing the project would lock in a false positive. The correct next step is diagnostic, not celebratory.
2. Design the measurement plan you would propose instead, naming the levels and the specific data sources.
Read the worked answer
Use a four-level plan anchored on behavior and results, with the knowledge test demoted to a supporting indicator.
Level 3 (behavior): structured observation of live or recorded calls using a rubric that scores whether the triage step is performed and performed correctly. Sample stratified by team and by tenure. Pair with tool-log compliance data (currently 60% triage use) as the continuous measure.
Level 4 (results): average handle time and escalation rate, tracked weekly with a baseline period and a comparison against non-adopting teams if any exist. Also track rework and first-contact resolution to catch gaming.
Level 2 (learning): keep the knowledge test but rewrite items to realistic case scenarios and validate against observed performance; report it as a leading indicator only.
Level 1 (reaction): retain but stop treating 4.4/5 as evidence of effectiveness.
Add a barrier analysis before any retraining: interview a sample of the 40% using the workaround and ask why. If the cause is tool usability or incentives, more training will not move the numbers, and the recommendation should go to the workflow owner, not to L&D. Present the plan to the director with the workaround data as the headline finding.