The Onboarding Trial That Cannot Randomize Individuals
A regional retail chain hires about 300 store associates a year and loses 38% within 90 days. HR has piloted a six-week mentoring program in two stores; the vice president wants a company-wide rollout if it "works." There are 40 stores in four regions, with 6 to 14 hires per store per year. Mentors are experienced associates on the same shift, so individual random assignment would create contamination: a control hire on the same shift could receive informal mentoring. Regional seasonality differs, with two regions peaking in summer and two in winter, and store managers choose start dates. The vice president proposes comparing 90-day turnover before and after rollout in all stores, using the pilot stores as evidence. You have one year and can delay rollout by store. What design do you recommend, and what would make the vendor's before-after claim unconvincing?
1. Recommend the strongest feasible evaluation design and identify the main threats it must handle.
Read the worked answer
The strongest feasible design is a store-level cluster-randomized waitlist trial. Randomize stores, not hires, because mentoring spills over within a shift. Stratify by region and by average hire volume, then assign roughly 20 stores to immediate mentoring and 20 to a waitlist that receives the program after six months. The outcome is 90-day turnover among hires, analyzed with a store-level random effect. This handles contamination and provides a counterfactual, but 40 clusters limits power; the design should be powered on the number of hires, not stores, and should not be sold as definitive if the effect is small. Main threats include selection if randomization is poorly executed, history and seasonality because regions peak at different times, maturation, differential attrition if high-risk hires leave early, and monitoring differences between immediate and waitlist stores. A stepped-wedge design with time fixed effects is the fallback if waitlisting some stores is politically impossible.
2. The vendor shows that the two pilot stores reduced 90-day turnover from 41% to 29% after mentoring began. Explain why this before-after comparison is weak evidence.
Read the worked answer
The comparison has no counterfactual. The two pilot stores may have been chosen because managers were already committed to mentoring, or because turnover was unusually high and likely to regress toward the mean. A 12-point drop could reflect history, such as a labor-market shift or a new scheduling policy, or maturation, such as hires settling in after the holiday rush. Attrition also matters: if the highest-risk hires leave before the 90-day mark, the remaining group looks better. The before-after design cannot separate mentoring from these explanations. A stronger claim would require comparable stores that did not receive mentoring during the same period, a pre-specified outcome, and evidence that the pilot and comparison stores had similar turnover and hiring profiles before the program. The vendor's result is a hypothesis, not proof of a 12-point effect.