A Triage Line Memo: Separating a Real Effect from a Measurement Artefact
A memo from the director of a district health office reads: 'Six months after launching our nurse-led telephone triage line, the share of emergency-department visits classified as non-urgent fell from 31% to 22%, and mean ED waiting time fell from 74 to 58 minutes. The triage line therefore reduced inappropriate ED use and eased crowding. We recommend extending the service to two more districts.'
Background facts the memo does not mention: the line handled 4,100 calls in six months while the ED recorded roughly 61,000 visits; in month three the ED revised its triage rubric, narrowing the definition of 'non-urgent'; in month one a walk-in urgent-care clinic opened two kilometres from the hospital; in month four the ED added two physicians to its evening roster; and callers advised not to attend the ED are not tracked afterwards. The director is asking the board to approve funding for two additional districts at the next meeting.
1. Identify the strongest single threat to the memo's causal claim and explain why it is stronger than the alternatives.
Read the worked answer
The rubric change is the strongest threat, because it attacks the outcome variable itself.
The headline measure - the share of visits labelled 'non-urgent' - is produced by the rubric. Narrowing the definition in month three mechanically reclassifies borderline visits into more urgent categories, so a fall from 31% to 22% can occur with no change whatsoever in who attends or why. That is a measurement artefact, and it sits directly on the quantity being reported.
The walk-in clinic and the two extra physicians are genuine confounders, but they load onto different outcomes: the clinic plausibly reduces attendances, and the physicians plausibly shorten waits. Neither explains a nine-point shift in labelling as directly as the rubric does. The clinic is also only two kilometres away, and a nine-point reallocation of a 61,000-visit population is a large effect to attribute to a nearby facility alone.
2. What evidence would most strengthen the claim that the triage line actually reduced non-urgent visits?
Read the worked answer
Track the callers, and remove the measurement shift.
First, a linked follow-up: for callers advised away from the ED, did attendance within 72 hours fall relative to a matched group of non-callers? Without this, the mechanism is asserted, not observed - the memo does not track callers at all.
Second, re-score a sample of pre-revision records under the new rubric. If the reclassified share accounts for most of the nine-point drop, the artefact explanation is confirmed; if the drop survives re-scoring, the artefact explanation weakens sharply.
Third, a comparison district without a triage line. If its non-urgent share stayed flat over the same six months, secular trends and the walk-in clinic become less plausible.
Scale is a warning sign throughout: 4,100 calls over six months is roughly 23 a day against about 335 ED visits a day. Even a large per-call effect is small relative to the population, which makes a nine-point population-level shift hard to attribute to the line.
3. Is the recommendation to extend the service to two more districts supported by this evidence?
Read the worked answer
No - the evidence is not yet strong enough to scale.
Of the three reported outcomes, two are contaminated (the classification share by the rubric change, the waiting time by the added physicians) and the third - actual attendance - is never measured. Extending to two districts would replicate an unverified mechanism at real cost, and it would make evaluation harder, because each new site brings its own co-interventions and its own rubric revisions.
A defensible recommendation would be: keep the line running, fix the measurement (freeze the rubric, add linked attendance follow-up, publish call and attendance volumes by district), and compare the pilot with a matched no-line district before committing.
Notice that the memo's own numbers are compatible with a genuine but much smaller effect. 'Keep and measure' is not a rejection of the service; it is a refusal to spend scale-up money on a before-and-after comparison with two simultaneous confounders.