Proving ROI for a Leadership Cohort With Confounded Data
A software company ran a six-month 'Leading at Scale' cohort for 24 engineering managers. Participants rated it 4.6 out of 5. Six months on, voluntary attrition among participants is 9%, against 14% for similar managers company-wide. But participants were nominated by vice presidents, and the VP of Engineering wants a dollar ROI figure before the next budget cycle. Two other changes landed in the same period: a compensation band adjustment and a new skip-level meeting cadence. You have survey access, HRIS data and roughly 30 analyst hours before the budget meeting.
1. Design the evaluation approach and explain how you will handle attribution.
Read the worked answer
Start by naming the confounds rather than ignoring them. Nomination by VPs means participants may already differ from the comparison group (higher potential, more sponsorship). The comp change and skip-level cadence are plausible alternate explanations for attrition.
Design within 30 hours:
- Comparison group: managers who were nominated but have not yet attended, or matched managers on tenure, level, and performance band. This is the strongest available counterfactual.
- Behavior measures (level 3): a short 360 or manager survey on the specific practices the cohort targeted - feedback frequency, delegation, escalation handling. This is closer to the intervention than attrition.
- Outcome measures (level 4): attrition, internal promotion rate, engagement scores, team-level indicators. Note that n=24 is underpowered for attrition; report it as a signal, not proof.
- Qualitative triangulation: six to eight interviews asking what changed and what else changed.
For attribution, present the attrition difference alongside the confounds, use the comparison group to narrow the estimate, and state plainly what cannot be isolated. Do not claim a causal ROI.
2. The VP wants one dollar figure. What do you actually present?
Read the worked answer
Give a range with visible assumptions, not a false point estimate. A single number invites a credibility failure the first time someone interrogates it.
Build a simple model:
- Cost side: design, facilitation, participant time, travel, backfill. This is the defensible part.
- Benefit side: avoided attrition valued at replacement cost per manager, with a sensitivity table. Show ROI if the attrition effect is zero, if it is three points, and if it is five.
- A plainer metric: cost per retained manager, or cost per participant who demonstrates the target behaviors. The VP can act on that even if the ROI range is wide.
Frame the decision, not just the number: what the next cohort would need to measure to justify scaling, and what evidence would cause you to recommend stopping. Offer a staged investment - a smaller cohort with pre-registered measures - which respects budget pressure rather than resisting it.
If asked to 'just give me the number,' say which assumption you would have to accept to produce it, and let the VP choose.
3. The data come back inconclusive. What do you recommend?
Read the worked answer
Report it as a finding, not a failure. Inconclusive results with n=24 and two confounds are an honest outcome, and overclaiming would damage your credibility for every future budget request.
Then do three things:
- Check implementation fidelity first. Did the cohort run as designed? Were managers' supervisors reinforcing it? A weak result may be an implementation finding, not a program finding.
- Look at the behavior data separately. Attrition may be too noisy, but if participants show clear gains on the targeted practices, that is a legitimate basis for a conditional recommendation.
- Recommend a cheaper, better-instrumented next step. Either a smaller cohort with pre-registered measures and a clean comparison group, or reallocating part of the spend to an area with stronger evidence, such as manager onboarding.
Give the VP a clear recommendation with the uncertainty stated: "Continue at reduced scale while we build the evidence" is a decision, and it is more useful than a hedged non-answer.