The Pilot That Looked Too Good
A team is planning a 12-month randomized trial of a text-message adherence intervention in 40 community pharmacies. The protocol budgets for a 6-month internal pilot at 4 pharmacies to test recruitment, the consent script, and the outcome-capture app. At the pilot's end, the steering committee sees enrollment at 140% of target, 96% retention, and an effect size of 0.55 SD on the primary outcome. The trial manager proposes folding the pilot sites into the main analysis and moving straight to the full trial with no protocol changes, arguing the pilot proves the design works. The statistician notes the pilot was not powered for the outcome and that pharmacy staff who volunteered for the pilot were the most enthusiastic in the network. The funder requires a documented go/no-go decision before releasing the remaining 70% of the budget.
1. What should the steering committee decide about including pilot participants in the main analysis, and why?
Read the worked answer
Pilot participants should generally be excluded from the main analysis, or their inclusion must be pre-specified and justified.
The core problem is that the pilot was conducted under different conditions: volunteer pharmacies, a small convenience sample, and staff with above-average enthusiasm. Folding these 4 pharmacies into the main trial means the intervention is being evaluated partly in a setting where implementation was atypical. That weakens the trial's external validity and can bias the estimate if pilot sites differ systematically from later sites.
A second issue is the risk of selection: if the pilot sites were chosen because they were likely to succeed, their outcomes are not exchangeable with those of newly recruited sites. Including them without adjustment mixes two sampling frames.
Practical guidance: decide now, before unblinding the main results, and write the rule into the protocol amendment. Common defensible options are (a) exclude pilot participants entirely, (b) include them only if the pilot was conducted under the final protocol with no changes and sites were randomly selected, or (c) include them with a pre-specified sensitivity analysis. What is not defensible is choosing after seeing that the pilot looked favorable.
2. The committee must make a go/no-go decision. What should it weigh, and what decision is most defensible?
Read the worked answer
The decision should be based on pre-specified progression criteria covering feasibility, not on the outcome effect size.
Weigh these:
- Recruitment: 140% of target is encouraging but comes from only 4 self-selected pharmacies; it does not establish that the remaining 36 will recruit similarly.
- Retention: 96% is strong, but again from enthusiastic sites.
- Outcome capture: the app worked in the pilot; confirm there were no data-quality problems before scaling.
- Effect size: 0.55 SD is essentially uninterpretable here. The pilot is not powered for the outcome, and a small enthusiastic sample inflates effects. Using it to justify proceeding or to size the main trial would be a planning fallacy.
- Funder constraint: the go/no-go must be documented.
The most defensible decision is proceed, but with conditions: keep the main trial, exclude pilot data from the primary analysis, expand recruitment to pharmacies selected without regard to enthusiasm, and pre-specify monitoring triggers (e.g., monthly recruitment and retention thresholds) that would pause the trial. Do not change the sample-size calculation based on the pilot effect. If feasibility criteria were pre-specified and met, proceed; if the pilot revealed implementation problems (e.g., consent burden), amend the protocol before scaling.