The Engagement Dashboard That Rewards the Wrong Behavior
A software company rolled out a 12-module onboarding program for new sales hires. Leadership asked for a dashboard and set a target: 85% of new hires should complete all modules within 30 days. Completion is defined as opening the module and clicking through to the end; there is no assessment. Six months in, the dashboard shows 91% completion, but sales managers report that new hires cannot explain the pricing model or handle a basic objection. Average time per module is 4 minutes, and module 7, on pricing, has the highest completion and the shortest average time. New-hire 90-day quota attainment has not improved. The VP of Sales wants to know why the training "isn't working" and is considering adding more modules.
1. What is wrong with the current measurement approach, and how would you redesign it to measure learning rather than clicks?
Read the worked answer
The dashboard is measuring exposure, not learning. Completion defined as opening a module and clicking to the end tells you the module was displayed, not that anything was understood. The 4-minute average and the fact that the pricing module, the hardest content, has the shortest time is a strong signal that people are clicking through without engaging. A 91% completion rate against an 85% target looks like success while the outcome that matters, the ability to explain pricing and handle objections, is failing. Adding more modules would scale the same problem.
I would redesign measurement in layers. First, replace click-through completion with evidence of learning: a short scenario-based knowledge check at the end of each module, with items that require applying the pricing model to a customer situation rather than recalling a definition. Second, add a performance task, such as a recorded or role-played objection-handling conversation scored against a rubric, at the end of the program. Third, track leading indicators that connect to the job: time to first independent quote, manager observation scores in the first 60 days, and 90-day quota attainment as the lagging outcome.
I would also keep an engagement metric, but as a diagnostic rather than a target. If a module is completed in 4 minutes, that should trigger a review of the module, not a celebration.
2. The VP wants to add more modules. How do you respond, and what do you propose instead?
Read the worked answer
I would respond directly: the data does not support adding content, because we have no evidence the current content is being learned. More modules would increase completion numbers and time cost without addressing the gap. I would bring the evidence: 4-minute average module time, the pricing module being fastest despite being hardest, manager reports of inability to explain pricing, and flat quota attainment.
Instead I would propose a redesign of the highest-value modules first, starting with pricing. That means cutting passive content, adding worked examples and practice with feedback, and requiring a scenario-based check before the module counts as complete. I would pilot the redesigned pricing module with the next cohort and compare performance-task scores against the current cohort.
I would also reset the dashboard target with the VP: completion stays as a hygiene metric, but the headline metric becomes performance-task pass rate and 60-day manager observation scores, with quota attainment tracked as the outcome. This reframes the conversation from "why isn't the training working" to "what evidence would tell us it is working, and what would we change if it isn't." I would ask the VP to sponsor that reframe, because changing the target without leadership air cover will look like moving the goalposts.