Sepsis Alert Rollout After Six Months of Data

Do the arithmetic on prevalence.

With 3.2% prevalence, sensitivity 0.88, and specificity 0.71, in a hypothetical 1,000 patients:

• True positives: 0.88 x 32 = 28.2
• False positives: 0.29 x 968 = 280.7
• PPV = 28.2 / (28.2 + 280.7) = about 9%, consistent with the observed 11%.

So the low PPV is not primarily a model defect - it is the arithmetic consequence of screening a low-prevalence population with a test that has moderate specificity. To raise PPV materially you must raise specificity, because prevalence is largely fixed.

What the numbers mean operationally

• 1,850 alerts in six months is roughly 10 per day; a 62% override rate at 11% PPV means most overrides are appropriate, but override fatigue is also how true cases get missed.
• Missing lactate in 28% of encounters means the model is running on incomplete inputs for a quarter of patients, which degrades both sensitivity and specificity in ways the original validation would not capture.
• Validation at one campus does not transfer: case mix, lab vendor, order sets, and nursing workflow all shift.

What to report: PPV, sensitivity, specificity, alert volume per clinician per shift, override rate, missed-case rate, time to antibiotics, and outcome measures - not accuracy alone.

Recommendation: no-go on the proposed date; conditional staged pilot instead.

Rationale

• Deploying an unvalidated model at sites with different workflow, lab interfaces, and case mix converts a local quality problem into a three-site problem and makes attribution impossible.
• Fixing the LOINC mapping is prerequisite work, not parallel work. Running a model with 28% missing lactate means any performance measured after go-live is uninterpretable.
• The dashboard request compounds the risk: publishing a measure that is not validated or locally reproducible invites external challenge.

Conditions for a revised timeline

1. Correct the interface mapping and confirm lactate capture above 95% for 30 days.
2. Recalibrate the threshold against local data to hit a target PPV agreed with the clinical leadership (for example 25%), accepting lower sensitivity if clinically tolerable.
3. Add a second-tier confirmation step to cut alert burden, and measure alerts per nurse per shift.
4. Run a 60-day silent or shadow mode at each new site, then a limited unit-level pilot.
5. Predefine stop rules and success thresholds before launch.
6. Validate performance by subgroup - age, sex, race, language, unit - to catch differential performance before it becomes a published equity problem.
7. Keep the dashboard internal until external validity is demonstrated.

Reframe what 88% means in this setting.

• Sensitivity was measured in the alert population at a single campus with a working interface. Twenty-eight percent missing lactate means the denominator at the new sites will differ, so 0.88 is not a portable number.
• Sensitivity and PPV trade off against the threshold. You can almost always buy sensitivity back by lowering the threshold, but that pushes alert volume up and pushback rises. The binding constraint is clinician trust, not raw detection.
• The failure mode of a low-PPV alert is not the false positives; it is the true positives that get overridden along with them. A 62% override rate is an early warning that the alert is being tuned out.
• Expansion multiplies governance risk: three sites, one metric, no shared validation, and a public dashboard.

What to offer the chair instead of a flat no: a defined pilot with dates, a named owner for the mapping fix, agreed thresholds, and a decision gate at 60 days with pre-set criteria. This preserves the clinical intent, protects the timeline, and gives the committee a defensible answer if the alert underperforms. Document the decision, the dissent, and the conditions in the committee minutes.