The Measurement Trap in a Well-Being Program

The claim is not supportable as causal from this design. Specific threats:

1. Attrition bias - 38% dropout with higher baseline exhaustion among non-completers means the remaining sample is selectively healthier. A meaningful share of the 13-point drop could be compositional, not real change. This is the largest threat.
2. Selection/rollout confounding - starting in the lowest-burnout departments biases the pooled post-score downward and confounds department with exposure.
3. Social desirability / demand characteristics - a dashboard visible to chairs creates pressure toward 'good' answers; self-report burnout under surveillance is a biased instrument.
4. Co-intervention - the month-8 documentation reduction is a plausible independent cause of reduced exhaustion. Any improvement cannot be attributed to the program alone.
5. Regression to the mean - modest, but relevant if departments were selected partly on high scores elsewhere.
6. Pre/post with no comparison group - secular trends (e.g., staffing changes) are uncontrolled.

Conclusion: the direction may be favorable, but the effect size is indeterminate and likely overstated.

Analyses:

• Attrition analysis: compare baseline scores of completers vs. non-completers; run a sensitivity analysis (e.g., multiple imputation or a worst-case/best-case bounding) to see how much of the 13-point drop survives plausible assumptions about non-completers.
• Stratify by department and rollout timing to separate program exposure from the documentation change.
• Interrupted time series if the dashboard has monthly data spanning before and after month 8 - this can partially separate the documentation effect from the program effect.
• Anonymity check: compare responses before and after the dashboard became chair-visible, if timing allows.

Message to leadership (plain, non-technical):

• 'The numbers moved in a good direction, and that's worth noting. But we can't yet say the program caused it, for three reasons: the people who dropped out were the most burned out, the program started where burnout was lowest, and a system-wide documentation change landed mid-program.'
• Recommend: retain the program (it's plausible and low-risk), but redesign the evaluation - independent anonymous survey channel, a comparison department, and pre-registered metrics for the next 12 months.
• Caution against publishing or marketing the 13-point figure; it would not survive scrutiny and could damage credibility.

Yes, and I'd explain why in terms of their interests, not methodological purity:

• Resource defense: the next budget cycle will ask whether the program works. A number that collapses under scrutiny is worse than an honest 'promising but unproven' - it invites cuts and undermines the well-being team's credibility.
• Physician trust: physicians are trained to read evidence. A program marketed on a biased pre/post will be discounted by the very people it's meant to reach, and the 'answered carefully' effect suggests they already suspect the measurement.
• Harm avoidance: if the dashboard pressures honest reporting, the program may be suppressing the signal it needs to help the most burned-out physicians - the non-completers are exactly the population at risk.

I'd offer a proportionate path: keep the program, fix the measurement (anonymous channel, comparison group, pre-specified metrics), and accept that rigor here is cheap relative to the program's cost. I'd also be clear about what I will and won't sign: I can support the program's continuation, but I won't endorse a causal claim the data can't bear.