Sepsis Alert Rollout and a Four-Point Mortality Difference
A nine-hospital system implemented a sepsis alert at three hospitals beginning in January 2020. The same quarter, all nine hospitals upgraded the enterprise EHR. Using 2018–2022 data, the analytics team identified 14,200 admissions meeting Sepsis-3 criteria. They defined exposure as 'alert fired at any time during the admission' and compared 30-day mortality: 18.2% in alert admissions versus 22.4% in non-alert admissions, p=0.01. Antibiotic timing also improved in the alert group. The three alert hospitals are the system's largest and have higher baseline case-mix scores. The team concluded that the alert reduced mortality by 4.2 percentage points and wants to submit the finding. A reviewer notes that patients who died within a few hours of admission could not have received the alert and that the EHR upgrade changed mortality coding at all sites.
1. What causal-inference threats does the 'alert fired at any time during admission' exposure definition create? State the likely direction of bias.
Read the worked answer
The exposure definition creates immortal time and time-varying confounding. To be classified as exposed, a patient must survive long enough for the alert to fire; patients who die very early are structurally classified as unexposed. This guarantees a period of survival in the exposed group and generally biases the mortality comparison toward showing benefit for the alert. The alert may also fire late in a deteriorating patient's stay, so 'alert fired' can be a marker of severity rather than a treatment; depending on timing, this can bias in either direction, but it makes the exposure non-comparable. Hospital-level confounding is substantial because the three alert hospitals are the largest and have different baseline case-mix. Secular trends are also confounded: the EHR upgrade occurred in all hospitals in the same quarter and could change coding, documentation, and mortality ascertainment. Differential measurement of sepsis criteria across sites and periods adds further risk. The key point is that the comparison is not between otherwise similar patients who did and did not receive an alert; it is between survivors long enough to be alerted and everyone else, across hospitals with different baseline risk and a concurrent system-wide change.
2. The team reports p=0.01 and a 4.2-point absolute mortality reduction. How should the committee interpret this result?
Read the worked answer
The p-value is not interpretable as reported because it ignores clustering. There are only nine hospitals, and patients within a hospital are not independent. With a plausible intraclass correlation, the effective sample size is much closer to the number of hospitals than to 14,200 patients, and the confidence interval for the mortality difference could easily include no effect. The 4.2-point absolute difference corresponds to a number needed to treat of about 24, which sounds impressive, but it is unadjusted for case-mix, hospital, and period. The result is also vulnerable to the immortal-time bias described above, which would inflate apparent benefit. A p=0.01 from a patient-level test in a hospital-level rollout is therefore weak evidence. The committee should not approve a causal claim. It can treat the finding as a hypothesis-generating signal and require a pre-specified analysis that respects the cluster structure, adjusts for hospital and time, and tests parallel pre-trends before any submission.
3. Propose a more defensible design and analysis plan using the available 2018–2022 data.
Read the worked answer
Reframe the evaluation as a hospital-level, staggered-adoption difference-in-differences or event-study analysis. Construct hospital-month panels of sepsis admissions, mortality, and antibiotic timing, with the pre-period beginning in 2018. Include hospital fixed effects and month or quarter fixed effects, and cluster standard errors at the hospital level, acknowledging that nine clusters provide limited power. Test for differential pre-trends between adopting and non-adopting hospitals; if pre-trends diverge, report that limitation rather than proceeding to a causal claim. Add negative-control outcomes that should not be affected by the alert, and a negative-control period before the EHR upgrade. For patient-level analyses, use a target-trial emulation with time-varying exposure and time-varying confounders, or a landmark analysis that starts follow-up at a defined point after admission, so that early deaths are not misclassified. Report both intention-to-treat at the hospital level and exposure effects with appropriate caveats. Conduct sensitivity analyses for unmeasured confounding, such as an E-value, and pre-specify subgroup analyses. The goal is an estimate that survives the timing and clustering problems, not a larger p-value.