The Pilot That Passed Too Easily
A hospital network is rolling out a new scheduling tool across 11 clinics. The pilot at one clinic ran for three weeks and reported 92% user satisfaction and a 15% drop in scheduling errors. On that basis, the programme director plans a big-bang rollout to the remaining 10 clinics in six weeks. A analyst notes that the pilot clinic had a dedicated on-site trainer, two extra administrative staff borrowed from elsewhere, and a scheduler with 12 years of tenure who manually corrected outputs before they reached staff. The other clinics have no trainer budget and average 3 years of scheduler tenure. The vendor's reference sites are all large urban hospitals.
1. Evaluate whether the pilot provides valid evidence for a big-bang rollout to the other clinics.
Read the worked answer
The pilot result is not transferable as presented. Three conditions inflated it: a dedicated trainer (support the other clinics will not have), borrowed staff (extra capacity that masked workload), and an expert scheduler silently correcting errors (so the 15% error reduction may understate the tool's true error rate). The pilot therefore measured the tool plus a favourable support environment, not the tool alone.
Big-bang multiplies risk: if the tool underperforms without the expert corrector, all 10 clinics fail at once with no unaffected comparison site. A staged or phased rollout, with the first two clinics instrumented and measured against the same metrics, would test the tool under realistic conditions and bound the blast radius. The reference sites also differ in scale and setting, so they add little assurance.
2. Design the rollout approach and the measures you would use to decide whether to continue past the first wave.
Read the worked answer
Move to a phased rollout: wave one of two clinics chosen to represent low and medium scheduler tenure, with a defined support package (remote trainer, not on-site) and no borrowed staff. Measure before and after on the same definitions used in the pilot: scheduling error rate measured on raw tool output before manual correction, staff time per schedule, and adoption (active users per week).
Set explicit continuation gates, for example: raw error rate no worse than baseline, time per schedule not increased, and adoption above an agreed threshold after four weeks. Include a qualitative check on workarounds, since silent correction is the failure mode the pilot hid. If a gate fails, pause and fix before wave two. This converts the rollout from a single bet into a sequence of informed decisions, and it gives the programme director evidence to justify either continuing or stopping.