Interoperability Interface Failing Silently
A regional clinic network sends lab orders and receives results from a hospital lab via an HL7 v2 interface. Over 3 weeks, the interface monitoring dashboard shows 'green' (no errors), but clinic staff report that approximately 12% of results never appear in the EHR. Investigation reveals the interface logs treat ORC/OBR segment mismatches as warnings, not errors, so unmatched messages are dropped after a retry. The lab vendor says the messages are valid per their spec; the EHR vendor says the clinic's order numbering is non-standard. The clinic sees ~1,400 results per week.
1. What is the failure mode here, and why did monitoring not catch it?
Read the worked answer
This is a silent data-loss failure at the interface layer, caused by a mismatch in message correlation logic (ORC/OBR identifiers) combined with monitoring that only tracks transport-level errors, not semantic or business-level failures.
Why monitoring missed it:
- The interface was technically 'up' - messages were received and acknowledged. Green status reflected connectivity, not delivery to the chart.
- ORC/OBR mismatches were logged as warnings, and the retry logic exhausted before escalation. Warnings don't page anyone.
- There was likely no reconciliation check between orders sent and results received per patient/order.
At ~1,400 results/week, 12% is roughly 168 results/week - a patient-safety exposure, not an IT nuisance. The 'whose fault' debate between vendors is a distraction; the clinic owns the patient-safety outcome regardless of spec interpretation.
2. Design a detection control that would have caught this within 24 hours, and describe how you'd validate it.
Read the worked answer
Detection control: order-to-result reconciliation
- Nightly (or hourly) job queries the EHR for orders with no matching result after an expected turnaround window (e.g., 48h for routine, 4h for stat).
- Separately, the interface engine exports counts of inbound result messages per order ID.
- A reconciliation report flags: (a) orders with no result, (b) results received with no matching order, (c) counts that diverge between the two sources.
- Route exceptions to a work queue owned by a named person, with escalation if unresolved in 24h.
Validation:
- Seed the environment with deliberately mismatched ORC/OBR messages and confirm they appear on the report.
- Run in parallel with production for 2 weeks and compare against manual chart audits.
- Define a threshold (e.g., >0.5% mismatch triggers investigation) and a named owner.
- After go-live of the control, do a 30-day look-back to confirm the historical 12% would have been flagged.
3. The vendors are pointing at each other. How do you drive resolution without waiting for them to agree?
Read the worked answer
Separate the technical fix from the blame conversation.
- Establish ground truth. Pull 20-30 affected message pairs and document exactly which field values diverge. This gives both vendors the same evidence.
- Define the interface contract in writing. Even if each side is 'valid per their spec,' the pairing must have an agreed correlation rule. Propose the clinic's order number as the canonical key and ask each vendor to confirm in writing.
- Assign interim mitigation to the clinic. While vendors negotiate, the interface engine can map/transform identifiers at the middleware layer - the clinic controls this and shouldn't wait.
- Escalate on a clock. Set a 10-business-day deadline with named vendor account managers; if unresolved, escalate to vendor executive sponsors and document in the vendor management file.
- Protect patients now. Implement a manual reconciliation worklist immediately so no result is lost while the fix is engineered.