Proving Impact When the Numbers Won't Cooperate
A retail chain rolled out a 6-hour customer-service training to 800 store associates across 40 stores over three months. Leadership wants proof it 'worked' before funding next year. You have: pre/post knowledge quiz scores (average rose from 58% to 81%), store-level customer satisfaction (CSAT) scores, sales per associate, and voluntary turnover. But CSAT also rose in stores that hadn't yet received training, two regions ran simultaneous promotions, and store managers varied wildly in how much they reinforced the training. You have no budget for a controlled study and two weeks to report.
1. What can you credibly claim about the training's impact, and what can't you claim?
Read the worked answer
Be precise about the evidence you actually have.
Can credibly claim:
- Learning occurred. The pre/post quiz gain (58%→81%) is a defensible Level 2 result, assuming the same assessment and comparable conditions.
- Reaction and completion if you have them (not stated, but check).
Cannot credibly claim:
- That the training caused the CSAT or sales changes. CSAT rose in untrained stores (a comparison group you didn't design but can use), promotions confounded sales, and reinforcement varied by manager. Attributing the outcome to training would be a causal overreach.
The honest framing: the quiz gain shows the training did its job at the learning level; the business-level story is suggestive but confounded. Say so. Overclaiming now damages credibility when someone later asks how you know-and leadership will ask.
2. How do you use the messy data you have to make the strongest defensible case in two weeks?
Read the worked answer
Mine the natural variation rather than pretending you had a clean experiment.
- Use the untrained stores as an accidental comparison. Compare CSAT change in trained vs. not-yet-trained stores over the same window. If trained stores improved more, that's suggestive (not proof), and it's far better than a raw before/after.
- Exploit the manager-reinforcement variation. Stores with high vs. low manager reinforcement likely differ in outcomes. If high-reinforcement stores show larger gains, that's both evidence the training can work and an actionable finding (reinforcement matters).
- Flag confounders explicitly-promotions, region, store size-so the analysis is transparent and can't be dismissed as naive.
- Lead with the solid Level 2 result, then present Level 3/4 as directional with caveats.
- Recommend the cheap fix for next time: stagger rollout by design (so untrained stores are a real comparison group) and standardize manager reinforcement. You don't need a lab study to get a defensible design.
3. Leadership wants a yes/no on 'did it work.' How do you frame your recommendation for next year's funding?
Read the worked answer
Refuse the false binary and give a decision-ready recommendation.
- What's proven: associates learned the material (58%→81%). That's a real result and worth stating plainly.
- What's likely but unproven: customer outcomes moved in the right direction, and the manager-reinforcement pattern suggests the training works when reinforced. Present this as directional with the confounders named.
- The recommendation: fund next year, but condition it on two low-cost design changes-staggered rollout so you get a real comparison group, and a standard manager reinforcement protocol. Tie the ask to what you'll be able to prove next year, not just what you hope.
Why this framing works: it's honest about uncertainty, which builds trust; it converts the messy data into an actionable insight (reinforcement drives results); and it gives leadership a reason to fund a better-designed next cycle rather than either rubber-stamping or canceling. A confident 'yes it worked' you can't defend is the fastest way to lose credibility and the budget.