Sepsis Alert Decay: A Unit-Conversion Defect Disguised as Model Drift
Riverside Health's sepsis early-warning tool runs every four hours on 900 inpatient beds, scoring patients and pushing alerts to the EHR. The vendor's FDA-cleared model had a validation sensitivity of 0.81 at the deployed threshold. Fourteen months after go-live, an internal review finds sensitivity of 0.68, a 22% increase in alerts per bed-day, and a clinician override rate of 61%. Nurse managers say the alert is 'ignored noise'. In March the hospital's laboratory switched to a new analyzer for lactate testing; results are now reported in different units and the interface maps them without conversion. The vendor released v4.0 in April under its predetermined change control plan; the hospital's contract gives it the right to deploy updates automatically. The CMIO wants to raise the alert threshold to cut noise. The sepsis committee wants the vendor to retrain on local data. The CIO wants to keep the tool because the state quality program ties reimbursement to sepsis bundle compliance.
1. Structure the diagnostic analysis. Distinguish the possible causes and say which evidence you would gather first.
Read the worked answer
Separate the observed symptom (more alerts, lower sensitivity, high overrides) from the cause. Three candidate causes behave differently in the data.
1. Data pipeline defect - the leading hypothesis The March analyzer change altered lactate units with no conversion. That is a coding error, not drift. Confirm by comparing pre- and post-March lactate distributions: a shifted or truncated distribution while clinical outcomes are unchanged is diagnostic. Check whether a plausible-but-wrong value now sits inside or outside the model's risk range.
2. Concept drift The relationship between features and sepsis changed (case mix, coding practice, treatment protocols). Test with performance stratified by cohort - sensitivity, specificity, calibration, AUC - before and after March.
3. Label drift and label lag The outcome definition or its ascertainment changed. Explicitly model the lag: alerts fired today are labelled against sepsis diagnoses appearing days later, so recent performance always looks artificially poor. Exclude the immature window before drawing conclusions.
Gather first
- Per-feature distribution comparison by month (PSI or KS) with clinical review of the top movers.
- Alert-level analysis: alerts per bed-day, positive predictive value, override rate by unit and shift.
- Chart audit of 50–100 overridden alerts and, critically, 50 septic patients who were not alerted. False negatives matter more than alert volume, and this is the evidence the noise complaint does not supply.
- v4.0 release notes against actual deployment date, to establish whether performance moved before or after April.
A single root cause is unlikely; the pipeline defect probably dominates, with concept drift contributing.
2. What interim action do you recommend - raise the threshold, suspend, or something else - and who decides?
Read the worked answer
All three proposals on the table are weakly justified as stated.
Test each option
- Raise the threshold: cuts alert volume and cuts sensitivity further. At 0.68 sensitivity, this is moving in the wrong direction without evidence the false positives are the dominant harm.
- Suspend: removes a safety net mid-quarter and creates its own preventable harm; defensible only if the tool is net harmful, which the audit of missed cases would establish.
- Do nothing: lets alert fatigue and missed sepsis accumulate while the committee debates.
Recommendation
- Correct the unit conversion immediately - fast, reversible, and it addresses the most likely cause.
- Re-run validation on the corrected pipeline and immature-window-adjusted data before touching the threshold.
- In the interim, keep the tool live with explicit clinician communication that its performance is under review, plus a manual lactate-aware screening checklist on high-acuity wards.
- Revisit the threshold only with the corrected performance curve in hand; pick the operating point against a stated sensitivity target, not against alert volume alone.
Decision rights The sepsis committee and medical executive committee decide, with the CMIO and patient safety; IT and the vendor do not. Document the decision, the rationale and the residual risk, and notify risk management and legal - missed sepsis cases during a degraded period are both a clinical and a liability exposure.
The judgement being tested: reimbursement tied to bundle compliance is not a reason to keep a degraded safety tool running unexamined. Say so plainly; it is the ethical answer, not merely the technical one.
3. How should the hospital handle the automatic v4.0 deployment and the demand for local retraining?
Read the worked answer
Automatic deployment is a change-control failure. The contract permits it, but governance should never have accepted the clause unqualified.
On v4.0
- Obtain the predetermined change control plan, the clearance documentation and the vendor's v4.0 validation evidence.
- Validate locally. A PCCP covers the vendor's intended changes for a specified population; it does not certify performance on Riverside's patients, its analyzer, or its case mix.
- Amend the contract: advance notice, a right to defer an update pending local review, structured evidence packages, an obligation to disclose performance signals, and support for local performance queries.
On local retraining The vendor will not train on Riverside data unless the contract provides for it, and a locally retrained model is effectively a new version needing its own validation and, depending on jurisdiction, its own regulatory pathway. Prefer local recalibration - recalibration intercept and threshold - which is cheaper, faster, and within the hospital's control, and reserve retraining for the case where recalibration demonstrably fails.
Institutionalise it
- Update the model inventory entry with version, validation date and known limitations.
- Set monthly monitoring: sensitivity proxy, alert positive predictive value, override rate, input drift - with numeric trigger thresholds.
- Name an owner for vendor performance.
If the vendor refuses notice-before-deployment, escalate to the AI governance committee as a supplier risk decision, and prepare the exit analysis before renewing.