Predictive Readmission Model Expansion Decision
A health system developed a machine-learning model to predict 30-day readmission for heart failure patients. The validation AUC is 0.78, but when deployed across three hospitals the alert fires for 62% of patients. The model uses 42 variables including prior utilization, laboratory values, and zip code. At Hospital A, the alert has been live for four months; nurses say it is always red and ignore it. At Hospital B, implementation stalled because of an EHR interface delay. At Hospital C, discrimination is lower (AUC 0.66) and the case-management team is smaller. The system quality council wants to expand the model to COPD next quarter. There is no plan for monitoring drift or bias. The vendor contract includes 99% uptime but no performance guarantees.
1. Assess the model's readiness for expansion and identify the most important risks.
Read the worked answer
Readiness: not ready. Validation AUC 0.78 is modest and does not establish clinical utility. Alerting on 62% of patients is a red flag: at that threshold, positive predictive value is likely low and alert fatigue is predictable. Hospital C's AUC 0.66 suggests distribution shift or different case mix. The model includes zip code, a proxy that can encode race and socioeconomic status; without bias testing, it may widen disparities. Hospital B has no implementation, so system-wide expansion is premature.
Most important risks: (1) poor calibration and threshold mismatch with case-management capacity; (2) alert fatigue and clinician override; (3) bias and lack of subgroup performance; (4) no drift monitoring or retraining trigger; (5) vendor contract lacks performance guarantees; (6) workflow not designed for the alert. Expansion to COPD would multiply these risks.
Recommendation: pause expansion; require local validation, calibration, subgroup analysis, and a workflow pilot before any new population.
2. Design a monitoring and governance plan for the model, including metrics, roles, and triggers for retraining or withdrawal.
Read the worked answer
Governance: assign a model owner (analytics), clinical owner (heart failure service), and safety/equity reviewer. Quality council reviews quarterly. Metrics: monthly alert volume, PPV, sensitivity at chosen threshold, calibration slope/intercept, AUC by hospital and subgroup (age, sex, race/ethnicity, language, insurance), override reasons, case-management enrollment, 30-day readmission, and data drift (PSI or KS). Workflow: human review before action; document alert rationale.
Triggers: retrain if calibration drifts beyond preset limits or AUC falls below 0.70 at any site; withdraw if subgroup harm appears, PPV falls below capacity threshold, or override rate exceeds 80% with no outcome benefit. Contract: add performance and audit clauses. Reporting: publish a model card and quarterly equity report. Do not expand to COPD until the heart failure model has two quarters of stable local performance and a bias audit.
3. How should the council respond to the proposal to expand to COPD next quarter?
Read the worked answer
Response: decline the next-quarter expansion, but do not reject it permanently. Reason: the heart failure model has not demonstrated local calibration, equity, or workflow benefit; Hospital B has not implemented it; Hospital C already shows lower discrimination. Adding COPD would spread limited analytics and case-management capacity and increase alert fatigue.
Alternative: set a staged pathway. (1) Stabilize heart failure: local validation, threshold adjustment, pilot at one hospital, equity audit. (2) Build monitoring and governance. (3) If heart failure meets prespecified criteria for two quarters, scope a COPD model with clinical input and a separate evaluation plan. (4) Consider simpler alternatives first, such as a validated rule or care-management referral. Communicate the decision with data, not authority; the council's credibility depends on disciplined sequencing.