Did the Diabetes Quality Improvement Project Work? Reading a Pre–Post Improvement Report
A family medicine residency clinic with 4 attendings and 900 adults with type 2 diabetes starts a registry with pharmacist-led outreach. At baseline, 61% of patients had an A1c <8%; at 12 months, 74% did. A subsequent chart audit finds that 22% of patients were 'excluded,' with reasons such as 'nonadherent' and 'no longer followed here.' The clinic also hired a new endocrinologist 6 months into the project, and the state expanded Medicaid coverage during the same period. One attending's panel improved from 58% to 79%; another's went from 47% to 50%. The program director asks whether the project 'worked' and whether it should be rolled out network-wide.
1. Delineate what conclusions these data do and do not support, naming the specific threats to validity at play.
Read the worked answer
Supported: the clinic can say that the proportion of included patients meeting the target rose from 61% to 74% over 12 months, and that the program was feasible to run. That is a description, not an effect estimate.
Not supported: that the registry/pharmacist program caused a 13-point improvement, or that a network-wide rollout will reproduce it.
Specific threats:
- Exclusion bias / selective denominator - 22% excluded for reasons like 'nonadherent' is enormous and not random. The patients most likely to fail are the ones removed, which mechanically inflates the remaining percentage. Unless the exclusion rule was pre-specified, applied uniformly, and auditable, '74%' is not comparable to '61%.'
- Secular trend / co-intervention - the new endocrinologist and the Medicaid expansion both plausibly improved control independent of the project. This is a pre–post design with no comparison group.
- Regression to the mean - the program was triggered by a low baseline; some rebound is expected regardless of intervention.
- Measurement/detection bias - patients engaged by outreach are also more likely to be tested; more A1c measurements can move a percentage in either direction depending on who gets tested.
- Small numbers / random fluctuation - the panel-level differences (58→79% vs. 47→50%) come from panels of roughly 150–250 patients; without confidence intervals you cannot tell signal from noise, and comparing two panels is not a test of the program.
What to say to the director: the project is promising, the data are insufficient to justify a network-wide claim, and the fix is a defensible measurement plan, not a bigger rollout.
2. Design the next 6-month measurement and improvement cycle: specify the measures, the analytic approach, and one concrete PDSA test.
Read the worked answer
1. Lock the denominator first (this is the highest-yield step). Write explicit, a priori, auditable inclusion/exclusion criteria (e.g., deceased, moved out of area, or transferred care - with a documented date). Ban soft exclusions like 'nonadherent.' Report both the raw and the adjusted denominators.
2. Build a balanced measure set:
- Outcome: % with A1c <8% (and <7% where appropriate); BP <140/90; LDL at goal; % with nephropathy screening and retinal exam.
- Process: outreach calls completed, pharmacist visits kept, statin prescribed, SGLT2/GLP-1 use in eligible patients with CKD or ASCVD.
- Balancing: hypoglycemia ED visits/admissions, staff overtime, patient-reported experience, cost per patient reached.
3. Analyze with time, not just two points. Monthly statistical process control or run charts with interventions annotated on the timeline. This distinguishes a real shift from ordinary variation and naturally addresses regression to the mean. Stratify by attending panel, insurance, language, and baseline A1c. If a rollout decision is needed, use a stepped-wedge or matched comparison clinic design rather than another single-arm pre–post.
4. One concrete PDSA. Plan: Practice makes no more than 10 outreach calls to patients with A1c >9% who have missed two visits. Prediction: at least 4 will attend a pharmacist visit within 3 weeks. Do: run for 2 weeks with a data sheet. Study: count completed contacts, visits kept, and reasons for failure. Act: if contact rate is low, test text-message outreach next cycle. Keep cycles small enough that a failure costs a week, not a year.
5. Report honestly - include the exclusions and the confounders in the write-up; that is what makes the project credible to the network.
3. The attending whose panel barely moved (47% to 50%) is defensive and says the data are 'cooked.' How do you handle this as the project lead?
Read the worked answer
Assume the concern may be partly valid and start there. Data skepticism from the person whose numbers look bad is not automatically bad faith - exclusion rules and denominator changes are exactly what he is worried about.
Steps:
- Audit the data with him, not about him. Walk through his panel list line by line: panel size, documented exclusions, patients who left, patients who never had a repeat A1c. Ask the analyst to produce a per-patient list. If a chunk of his denominator is patients who cannot be reached, the measure - not the physician - is the problem.
- Look for legitimate structural explanations. Panel composition (more uninsured, more语言 barriers, higher baseline A1c), support-staff ratios, schedule template, referral access, and burnout are all real and modifiable. Ask him what he believes is in the way.
- Recruit him as co-designer. Give him ownership of one element (e.g., how outreach is offered to his patients). People defend a measure they helped build.
- Never present panel-level performance in a punitive public forum. Share data in a confidential, improvement-oriented format; celebrate the process measures, which everyone can control, rather than only the outcome.
- Set a shared, time-bound goal with a re-audit date, and offer coaching or a colleague pairing.
- Escalate only for genuine patient-safety or competency issues - through the department chair or peer-review process, not through the QI dashboard. Disagreement about a metric is not a professionalism problem; refusing to review data or obstructing documentation may be.
- Debrief yourself: if one panel is flat while another rises 21 points, ask whether the intervention is actually reaching all patients or just the easiest ones. That is the improvement question the director actually needs answered.