Three Sites, One Average, and a Board That Wants to Expand

The 68% is a within-group pre-post comparison with no counterfactual, and the aggregate conceals the most important finding.

• No comparison condition. Intake-to-exit change cannot separate program effect from maturation, local hiring conditions, seasonality, or regression to the mean.
• Regression to the mean is a serious candidate. Intake is typically triggered by a crisis or a low point; exit is measured later. Some of the 41%→68% shift is expected with no program at all.
• Site heterogeneity. A (+30) and B (+30) look like something happened; C (+6) looks like nothing did. Averaging them produces a number that describes no site in the report.
• Attrition and missingness. The 11 uncounted exits at C are coded as failures. That is an assumption, not a measurement, and it was made after the fact.
• Measurement independence. If the staff who deliver coaching also assess the benchmark, the outcome is not blind to who delivered it.
• Unreported fidelity plus 38% turnover means you cannot say what dose participants actually received.

Causal language is defensible only within Sites A and B, and only with a comparison.

• Report missing as missing, not as failure. Present three figures: complete-case (68%), intention-to-treat with missing counted as failure, and a sensitivity bound with missing counted as success. If the conclusion flips depending on which you choose, the honest statement is that the effect is uncertain within that range.
• Test whether the missingness is informative. Compare the 11 to completers on baseline benchmark, referral source, distance from site, and any risk indicators. If they entered lower-functioning, calling them failures overstates the loss; if they were similar, the failure assumption is more defensible.
• Report Site C separately and in full as a finding about implementation, not as an outlier to be trimmed. Name the cause - total front-line replacement in month 5, no exit data for 11 clients - and use it to argue that the model requires staffing stability. That is precisely the risk the two new sites would be taking on.
• Do not quietly drop C to improve the aggregate, and do not let it be the only caveat either. Per-site results should be the primary table, with the aggregate clearly labeled as an average across heterogeneous sites.

Before endorsing expansion, I would require the following in writing:

1. Per-site results as the headline, with the aggregate explicitly labeled as an average across sites that behaved very differently.
2. Fidelity and dose data from A and B for the past twelve months. If the organization cannot produce it, the model is not documented well enough to replicate.
3. A stronger design at the new sites - staggered start (stepped-wedge) or matched comparison, with an analysis plan filed before enrollment begins.
4. A staffing stability plan and a turnover threshold, given 38% turnover and what happened at C.
5. An exit-data system with a ≥90% completion target and a missing-data protocol specified in advance.
6. Independent or blinded outcome assessment, at minimum for the benchmark.
7. Cost per participant who crosses the benchmark - the number the board actually needs to make a scale decision.
8. Corrected fundraising language now. The 68% claim should be pulled or rewritten before it is printed again; a claim repeated in fundraising and contradicted by the organization's own evaluation is a compliance and reputational exposure, not just an accuracy problem. I would say that to the board directly, in the same meeting where I recommend the evaluation plan.