The Readmission Model Nobody Asked For (Population Health)
A 420-bed nonprofit hospital holds a value-based contract in which 30-day unplanned readmissions trigger penalties; last year those penalties totaled $1.4M against a baseline readmission rate of 14.2%. The VP of Population Health asks your team to "build an AI model that predicts readmissions and get us to 10%." Investigation shows the only intervention on offer is a transitional-care nurse call placed within 48 hours of discharge, and the team has 62 call slots per week against roughly 240 weekly discharges. Nurses currently choose patients using a LACE score above 10 plus clinician referral. The EHR holds 12,400 discharges from the past two years with demographics, labs, prior utilization, discharge disposition, and diagnosis codes. The VP expects a monthly report of model AUC and wants a pilot live in ten weeks. The care management director is skeptical, noting that a previous vendor risk score "just told us everyone was high risk."
1. Rewrite the VP's request as a decision problem, and state the primary success metric you would propose in place of AUC. Justify both choices.
Read the worked answer
The VP has described a tool, not a decision. Unpacked: about 240 patients are discharged each week, 62 receive a call, and the call is the only lever. The operative decision is therefore a ranking-under-capacity problem - which 62 patients occupy the slots - and the model is an input to that decision, not the deliverable.
The reframing changes three things:
- Population. Only patients plausibly helped by a call belong in the ranking. Planned readmissions, hospice discharges, and transfers out should be excluded, or the model spends its top slots on patients no call can help.
- Metric. Because slots are scarce and fixed, the question is "of the 62 we contact, how many would have been readmitted anyway?" Precision at 62 per week (or lift over the LACE rule within the top 62) answers that. AUC averages discrimination across the entire score range, including the ~75% of patients who will never receive a slot.
- Outcome. The business cares about readmissions avoided per nurse-hour, tied to the $1.4M penalty pool.
I would propose readmissions avoided per 100 calls relative to the LACE baseline as the primary metric, keeping AUC as a diagnostic only. That also lets the pilot's value be stated in dollars.
2. What must be true about the prediction target and its timing for this model to be useful? Name at least three specification choices you would make explicit and defend each.
Read the worked answer
The target must be defined at a moment when the prediction is actionable and the inputs actually exist. Three choices I would force into writing:
- Outcome definition. 30-day unplanned readmission to this hospital or any hospital visible in the HIE, with CMS-style exclusions for planned returns. Without the exclusions the model learns scheduled procedures; without the HIE look-through it under-counts and rewards patients who simply went elsewhere.
- Prediction timestamp. Score at discharge-order time, not at admission. A discharge-time score can incorporate medication reconciliation, discharge disposition, and follow-up scheduling, and it aligns with the 48-hour call window. An admission-time score may be stale by day four and may omit the most predictive information.
- Treat competing outcomes explicitly. Death and transfer out are not "no readmission" in any meaningful sense; coding them as negatives teaches the model that the sickest patients are safe.
One more: freeze the feature set as of the scoring timestamp. A field like "follow-up appointment scheduled" is only legitimate if it is populated before the call is placed.
3. The care management director asks how you will know the model beats the existing LACE rule. Describe the evaluation design, including what you would not do.
Read the worked answer
The only credible comparison is head-to-head on the same patients under the same capacity constraint.
Design. Run a silent-mode period of 8–12 weeks in which the model scores every discharge while nurses keep using LACE. Then compare the two rankings on (a) precision at 62 per week, (b) list overlap, and (c) the reallocation effect - how many patients the model swaps in who were in fact readmitted. Silent mode also delivers a local calibration check and surfaces pipeline errors before anyone depends on the output.
What I would not do. I would not randomize patients to "model vs. no call." That answers a different question (does the call work) and is hard to defend when LACE-based outreach is the standard of care. If randomization is possible, randomize the selection rule at the margin where the model and LACE disagree - both arms are then standard-of-care selections, and the contrast isolates ranking quality.
Power. With 62 slots per week, readmission counts are small. Budget at least a quarter before claiming a 10–15% improvement; report confidence intervals and pre-register the metric so the result is not chosen after the fact.