The Pump That Read the Wrong Order
Your 350-bed hospital runs 220 large-volume infusion pumps integrated with the EHR through a vendor middleware gateway. At 02:10 the gateway's TLS certificate expired. The pumps failed safe - they dropped out of auto-programming and reverted to manual mode - but nothing alerted anyone; the pumps simply stopped receiving orders. At 06:40 a nurse programmed a heparin infusion from the pump's local drug library using a concentration that differed from the active order, overrode the library warning, and infused for 50 minutes before pharmacy caught it. At 08:00 the pharmacy director pulls a dashboard showing 94% of infusions auto-documented and says integration is fine. Nursing says "the pumps are broken."
1. Reconstruct the failure. Why did the 94% auto-documentation number coexist with a broken integration?
Read the worked answer
Two different metrics were being read as one.
- The integration path has several distinct stages: order selection → patient/order matching → auto-programming at the pump → auto-documentation back to the EHR. These can fail independently.
- When the TLS certificate expired, the gateway could no longer authenticate to the pumps. Auto-programming stopped. But the pump still lets a nurse scan and manually program, and manual programming still generates a documentation record - so auto-documentation can stay high while auto-programming is at zero.
- A 94% dashboard figure also invites questions about its denominator and window: is it today, this week, this month? Does it count scans as documentation? A rolling window would barely move in six hours.
The fail-safe behavior is actually correct and desirable - a pump that keeps delivering when the network dies is safer than one that stops. The design defect is that the fallback was silent: no alert to the biomed/IT on-call, no banner on the pump, no flag to nursing that orders are no longer flowing. Fail-safe without fail-visible is not safe.
Nursing's "the pumps are broken" and pharmacy's "integration is fine" are both wrong, and both are describing real symptoms of the same event.
2. Give the immediate response sequence for the next four hours, including who must be told and what must be verified before integration is turned back on.
Read the worked answer
First hour - contain and scope:
- Query the gateway and pump logs to establish exactly when auto-programming stopped and how many pumps and patients were affected. Get the count before you speculate.
- Notify nursing leadership, pharmacy, and the IT/clinical informatics on-call. State plainly: auto-programming is unavailable, manual programming with independent double-check is required.
- Require order verification against the MAR and an independent double-check for all high-alert infusions (heparin, insulin, vasopressors, opioids) until further notice.
- Enter the heparin event into the patient safety reporting system now, and trigger a clinical lookback: aPTT, platelet count, bleeding assessment, and the actual delivered dose versus ordered dose.
- Renew/replace the certificate, but do not restart the middleware on the assumption that renewal fixed everything.
Before restoring integration - verify:
- Certificate chain valid and system clocks synchronized across gateway, pumps, and EHR (a clock skew will break authentication again).
- Each pump re-enrolled and reporting; not just the gateway showing green.
- A test order pushed end-to-end from the EHR to a non-patient pump and documented back.
- Confirmation that no stale or queued orders are waiting to flood the fleet on reconnect.
If any of that is unverified, integration stays off and manual workflows stay in place. Manual with double-checks is slow but safe; premature reconnection can push a queued order to the wrong patient.
3. What design and monitoring changes would prevent a silent certificate expiry from reaching a patient next time?
Read the worked answer
Monitoring and alerting
- Certificate expiry monitoring with escalating alerts at 90/30/7/1 days, owned by a named role - not a shared mailbox.
- Synthetic end-to-end transactions (order → pump → documentation) run periodically, alerting on failure. Health checks that only ping the gateway would have shown green all night.
- An explicit alert whenever any pump transitions from integrated to manual mode. That single alarm would have caught this at 02:10.
Design
- Display a persistent, visible banner on the pump and a status flag in the EHR when integration is down, so the clinician at the bedside knows the order is not linked.
- Keep the local drug library aligned with the formulary and make concentration mismatches hard stops rather than overridable warnings for high-alert drugs. The nurse's override was a human factor, but the system invited it.
- Require BCMA patient matching to be part of the manual fallback workflow, not skipped because the pump is offline.
Metrics that would have surfaced it
- Report auto-programming rate separately from auto-documentation rate, and trend both.
- Track drug-library override rates by drug and by unit; a heparin override should be a review item, not a statistic.
Process
- Treat certificate and integration changes under formal change management with a rollback plan and a nurse-facing communication step, not as a routine IT patch.