Reading a Quality-Improvement Summary: Association or Cause?
A quality-improvement coordinator presents this summary at a clinic meeting:
"In the first quarter, the two clinics that adopted automated text-message reminders saw no-show rates fall from 19 percent to 14 percent. The two clinics that did not adopt reminders fell from 18 percent to 17 percent over the same period. During the quarter, the two adopting clinics also began offering two evenings of extended hours each week. Average time to the next available appointment rose at all four clinics, from 9 days to 13 days. No-show data were pulled from the scheduling system for all new and follow-up appointments. Reminder opt-out rates at the adopting sites were 31 percent. Reminders were sent 48 hours before each appointment."
Administrators propose expanding text reminders to the remaining two clinics next quarter. The coordinator's slide is titled: "Reminders Cut No-Shows by Five Points."
1. State the most defensible conclusion the summary supports. Then explain why the slide's title is not that conclusion.
Read the worked answer
Defensible conclusion: no-show rates fell more sharply at the two clinics that adopted reminders (five points) than at the two that did not (one point), but the summary cannot separate the effect of reminders from the effect of the extended evening hours introduced at the same clinics in the same quarter.
Why the title overreaches:
- Confounding. Two things changed at once at the adopting sites. Either could explain the drop, and the summary provides no way to apportion them. The comparison group controls for nothing that happened only at the adopting clinics.
- The five-point figure is a raw difference, not an effect. The non-adopting clinics also improved by one point, so the gap attributable to something different at the adopting sites is four points, and that something is unidentified.
- 31 percent opted out. Nearly a third of patients at the adopting sites received no reminder at all, so "reminders" did not reach everyone whose no-show behavior is in the numerator.
Rule being tested: a difference between groups is an association; calling it "cut" asserts causation, which requires ruling out the rival explanation.
2. What single additional piece of information would most strengthen a causal claim, and why is it stronger than simply collecting another quarter of the same data?
Read the worked answer
The strongest addition: appointment-level data from the adopting clinics comparing no-show rates for patients who received a reminder with rates for patients who opted out, in the same quarter.
Why this beats another quarter of the same summary: the opt-out group exists inside the same clinics, at the same time, under the same extended hours. Comparing reminded against opted-out patients holds the extended-hours change constant, which is precisely the variable that ruins the between-clinic comparison. The natural experiment is already sitting in the data.
A second acceptable answer: a look at no-show rates at the adopting clinics in the same quarter of the prior year, to test whether the drop is a seasonal pattern rather than an intervention effect.
What would not strengthen the claim: repeating the same two-clinic comparison for a second quarter, because the confound - extended hours - persists in every repeat. More data of the same design does not fix a design problem; it only makes the same number more precise.
3. The summary states that average time to the next available appointment rose from 9 to 13 days. How does this detail complicate the story, and what mistake would a reader make by ignoring it?
Read the worked answer
How it complicates the story: the summary's implied goal is to get patients to their appointments. The intervention appears to have improved attendance, yet access got worse at every clinic, including the two that changed nothing. A four-day increase in wait time is a real cost to patients, and it may itself influence no-shows - a patient facing a 13-day wait has more time for the appointment to become inconvenient. The detail is therefore not neutral background; it is a competing explanation the summary never addresses.
The mistake in ignoring it: treating a single favorable number as the whole result. A reader who takes only the five-point drop as the finding has accepted the coordinator's framing without noticing that the same document reports an offsetting trend and that the trend is uniform across all four sites - which, notably, is the one pattern in the summary that the reminder rollout cannot explain at all.
Reading skill involved: details that undercut the main claim are usually stated plainly, not hidden. They are missed because the reader is tracking the number the writer emphasized.