Sepsis Alert at 4 a.m.
Mercy Valley Health is a four-hospital system on a single EHR. Six months ago it deployed a vendor sepsis prediction model in two ICUs and one 36-bed medical-surgical unit. The model fires on 12% of admissions; positive predictive value is 18% and sensitivity 71% against the local definition. Nurses dismiss 34% of alerts within 30 seconds. Lab interface latency averages 45 minutes, and the model's features include lactate and antibiotics that arrive after the prediction window. The vendor retrains quarterly on a national dataset and does not provide drift metrics. Sepsis mortality fell 1.2 percentage points in pilot units, but a concurrent antibiotic stewardship program launched in month two. The contract renews in 90 days; the CMIO wants to expand to six more units; the CNO says the alert is 'noise.' There is no local validation, no monitoring dashboard, and no documented clinical owner.
1. Assess whether the pilot shows the model works. What analyses and evidence are needed before renewal?
Read the worked answer
The pilot does not yet show the model works. The 1.2-point mortality decline is confounded by a concurrent stewardship program, has no control group, and is small enough to be noise. The operating metrics are more diagnostic: a 12% alert rate with 18% PPV means roughly one true case for every 5.5 alerts; 34% dismissed in under 30 seconds indicates the alert is not arriving at a moment or in a form that changes action.
The 45-minute lab latency is a validity problem, not just an IT problem: if lactate and antibiotic features arrive after the prediction window, the model may be partly retrospective and its apparent sensitivity is not achievable prospectively. Before renewal, require: local validation on at least 12 months of data with AUROC, calibration by unit, PPV at the deployed threshold, and subgroup performance; a time-to-alert analysis; a leakage check on each feature's availability at scoring time; and an interrupted time-series or matched comparison for outcomes, with the stewardship program modeled. Also measure process outcomes-time to antibiotics and bundle compliance-because that is the mechanism by which sepsis models help.
If local AUROC is below 0.70 or calibration is poor, the vendor's national performance is not transferable. The CNO's 'noise' judgment is a legitimate safety signal, not resistance to change.
2. What lifecycle and monitoring controls must exist before expansion? Specify owners and thresholds.
Read the worked answer
Before any expansion: name a clinical owner (co-owned by CMIO and CNO) and a technical owner; document the intended use and the population; and set a local threshold based on the PPV/alert-burden tradeoff. Fix the interface latency or exclude features that are not available at scoring time. Redesign alert routing to the role that can act, with a single alert per patient per window and suppression after acknowledgement. Run a silent trial of the retuned model for 30 days before go-live.
Monitoring must be prospective and owned: weekly alert rate, PPV, override rate, time-to-antibiotic, sepsis bundle compliance, and outcome measures; monthly calibration and AUROC where labels are available; quarterly subgroup review. Thresholds: alert rate ≤8% of admissions, PPV ≥25%, override rate <20%, calibration slope 0.8–1.2, and AUROC ≥0.75. Any breach triggers review, not automatic retraining.
Model change control: vendor retraining must be versioned, validated locally before deployment, and communicated. Incident process: a missed sepsis case or a sustained PPV drop is a safety event, not a ticket. Data quality: monitor missingness of lactate and antibiotics; if missingness rises, pause. Finally, document the human-factors work: nurse co-design, alert text, and training. Expansion without these controls would multiply the current alert burden by six units and make the CNO's concern the correct risk position.
3. Recommend a decision and the contract terms to negotiate.
Read the worked answer
Recommend a 90-day conditional extension limited to the three current units, with no expansion to six more. Use the 90 days to complete local validation, fix latency and threshold, run the silent trial, and stand up the monitoring dashboard.
Negotiate: local validation rights and access to the vendor's validation data; drift and performance reports at least quarterly; notification of model changes and the right to reject a version; a service-level agreement on interface latency; audit rights; clarity that the vendor does not use our patient data for retraining without approval; and termination for sustained performance failure. Tying a portion of fees to outcomes is difficult in clinical contracts and may create perverse incentives; better to use a pilot extension with defined exit criteria and a price hold.
If the vendor will not provide drift metrics or local validation support, do not renew; the system cannot be governed. Communicate to the CNO and nursing leadership that expansion is paused and why, and give them a role in threshold setting. Communicate to the board in terms of alert burden and safety, not model elegance. The CMIO's expansion proposal should be deferred, not rejected; if the 90-day evidence is strong, expand in two units at a time with a repeat gate.