The Program That Looks Great on Paper
You lead a care management program for high-risk patients with COPD. Your dashboard shows a 22% reduction in 30-day readmissions versus last year, and leadership wants to expand it. But when you dig in: the program enrolled 180 patients, but 40 were excluded after enrollment for 'not high-risk enough'; the comparison is last year's all-comers, not a matched group; readmissions are counted only at your own hospital, though two competitors are nearby; and the 6 patients who died were counted as 'not readmitted.' Staff also report they are spending most time on the easiest-to-reach patients because the hardest ones miss appointments.
1. Identify the measurement problems and explain how each distorts the result.
Read the worked answer
Four distortions, each inflating the apparent effect:
- Post-enrollment exclusion (40 patients): removing 'not high-risk enough' patients after they were enrolled selects a healthier or higher-risk mix post hoc. This is selection bias - the denominator no longer reflects who the program actually served.
- Historical all-comers comparison: last year's patients differ in case mix, seasonality, and concurrent initiatives. Without matching or a concurrent comparison, you cannot attribute the change to the program.
- Single-hospital readmission counting: readmissions to nearby competitors are invisible, so the program looks better the more it shifts care elsewhere. This is surveillance bias.
- Deaths counted as non-readmissions: patients who die cannot be readmitted. Counting them as successes makes a program that fails the sickest patients look effective - a competing-risk problem.
Judgment point: each flaw biases in the same favorable direction, which is a red flag that the result is measurement, not impact.
2. Staff gravitate to the easiest-to-reach patients. How does this interact with the outcome data, and what do you do?
Read the worked answer
This is cream-skimming, and it compounds the selection bias: the program's measured success partly reflects serving lower-risk patients while the highest-risk patients (who miss appointments) go unserved. The readmission reduction may be real for the served group but is not generalizable to the population the program claims to help.
Actions:
- Measure reach and equity: track the proportion of eligible high-risk patients actually engaged, and stratify outcomes by risk tier.
- Redesign outreach for the hard-to-reach: proactive contact, home visits, transportation support, and flexible scheduling - the same barriers you'd address clinically.
- Change the incentive: if staff are evaluated on visit volume, they will optimize for easy visits. Evaluate on engagement of the highest-risk cohort.
Reasoning standard: a program's value is engagement × effect, and engagement of the hardest patients is where the effect lives.
3. Leadership wants to expand now. What do you recommend, and what would a defensible evaluation look like?
Read the worked answer
Recommend a pause on expansion until the evaluation is fixed. Expanding on a biased result risks scaling something that doesn't work and locking in the measurement problems.
A defensible evaluation would:
- Predefine the cohort (risk criteria set before enrollment) and analyze by intention-to-treat, including everyone enrolled.
- Use a concurrent comparison - matched controls, a stepped-wedge, or difference-in-differences - rather than last year's all-comers.
- Capture readmissions across all area hospitals (claims or HIE data).
- Handle deaths correctly: report readmission and mortality together (composite or competing-risk analysis), not deaths as successes.
- Report reach, equity, and patient-reported outcomes alongside readmissions.
Judgment point: the ethical and professional obligation is to report what the data actually show, even when leadership wants a green light. Present the corrected analysis and a phased expansion with monitoring.