The 18 Percent Claim and the Surveillance Risk
You lead learning analytics at a 1,500-employee SaaS company. An eight-week sales onboarding program combining product training, call shadowing, and certification launched 11 months ago. The VP of Sales has told the executive team that the program produced an 18 percent higher quota attainment in the first six months. The underlying data cover 120 new hires: 80 completed the new program and 40 went through the legacy onboarding. Assignment depended on hiring cohort and territory. Twelve records are incomplete. Certification pass rate is 88 percent, and manager coaching hours vary from zero to six per month. The company now wants to expand analytics to track post-training behavior using calendar and call data, flag 'low performers' to managers, and pilot the approach in Germany, where a works council represents 200 employees and GDPR applies. You must present to the executive team in three weeks and have one analyst supporting you.
1. How do you assess the VP's 18 percent claim, and what can you responsibly tell the executive team?
Read the worked answer
First define the outcome. Eighteen percent higher quota attainment compared with what - legacy onboarding, the prior year, or the plan? Check whether the measure is quota, revenue, time to first deal, or ramp time, and whether the 12 incomplete records change the result.
Then examine selection. The 80 participants were not randomly assigned; assignment followed hiring cohort and territory. Newer cohorts may have better territories, different managers, or a changed product. A raw comparison confounds the program with who received it.
Do what the data allow. Compare participants with legacy hires on tenure, territory potential, manager, and start quarter. If sample size permits, use a matched comparison or a regression with those controls; if not, present ranges and say so. Check implementation variation: new hires with zero coaching hours versus six may differ more than program versus legacy.
Tell the executive team: 'The early pattern is consistent with a positive effect, but the design cannot separate the program from territory and cohort effects. I would not repeat 18 percent as proven.' Then propose a stronger next test: stagger start dates or use a waitlist so comparison groups are formed by timing, and pre-register the measures.
Protect credibility. Overstating now makes the next honest result easier to dismiss.
2. What do you recommend on the expanded tracking of calendar and call data and the German pilot?
Read the worked answer
Separate learning analytics from employee surveillance. Tracking post-training behavior can be legitimate for program improvement; flagging individuals as 'low performers' to managers is a different purpose with higher legal and trust risk.
Do the groundwork in Germany first. Consult the works council before designing the pilot, not after. Complete a data protection impact assessment, define the legal basis, minimize data, set retention limits, and document who sees what. GDPR and German co-determination practice make individual monitoring difficult without agreement.
Redesign the default: aggregate dashboards at team level, voluntary coaching opt-in, and no individual labels. Use CRM outcomes and self-reported confidence instead of calendar surveillance where possible. If individual flags are essential, define due process, context, and a right to respond.
Pilot narrowly in one country and one team with clear evaluation criteria, including employee trust and participation, not only predictive accuracy. Pause if the works council does not agree.
Advise leadership on the risk: a sales analytics program that feels like monitoring can reduce the data quality it depends on. The ethical choice and the practical choice point the same way here.
3. How do you structure the 30-minute executive presentation?
Read the worked answer
Lead with the decision, not the methodology. One slide: 'We recommend extending the program, but not repeating the 18 percent claim; here is the evidence and the next test.' Executives need to know what you are asking them to decide.
Then three points: what the data show, what they cannot show, and what you will do about it. Use a simple chart with confidence intervals or a range rather than a single bar. State the confounders in plain language: territory, cohort, manager coaching.
Pre-brief the VP of Sales and the CHRO. Give the VP a face-saving framing: the program looks promising and the next cohort design will let us prove it. Do not surprise him in the room; that turns a measurement issue into a political one.
Include the privacy item as a separate decision with its own risks, timeline, and works council dependency. Do not bury it in the analytics discussion.
Close with owners, dates, and what would change your mind. A good executive presentation is a decision memo, not a lecture on causal inference.