Department Quality Initiative: A Measure That Would Mislead
You lead a pediatric outpatient therapy team. Administration asks you to adopt a new quality metric: 'percent of children who improve at least 5 points on a functional mobility measure by discharge,' to be compared across therapists and used in annual reviews. The measure is validated for children ages 2 to 12 with neuromuscular conditions. Your caseload includes infants under 1 year, children with progressive conditions, and a large group of children with chronic pain and hypermobility. Last year your team discharged 210 children; 38 were under age 2, 22 had progressive diagnoses, and 41 had chronic pain. Therapists have begun declining complex referrals since the metric was announced. You have 6 months before the metric is scheduled to go live.
1. What is wrong with this metric as designed, and what would you propose instead?
Read the worked answer
Problems:
- Out-of-scope population. The measure is validated for ages 2–12 with neuromuscular conditions. Applying it to infants under 1 year and to children with chronic pain/hypermobility is a validity violation; the score change may not mean what the metric assumes. Roughly 79 of 210 discharges (38 infants + 41 chronic pain) fall outside the intended population.
- Progressive conditions. For 22 children, stability or slower decline may be the appropriate outcome. A metric that rewards only improvement penalizes clinically correct care and can incentivize inappropriate goal-setting.
- Attribution and case mix. Comparing therapists without risk adjustment compares caseloads, not skill. Therapists with complex or progressive patients will look worse.
- Gaming and access harm. Therapists are already declining complex referrals - the metric is producing the opposite of the intended effect and creating an access/equity problem.
- Use in annual review. Tying an unadjusted, out-of-scope measure to employment decisions amplifies all of the above.
Proposed alternative:
- Restrict the measure to the validated population; use a separate, appropriate measure for infants and for pain/hypermobility.
- Add goal-attainment scaling or a validated participation measure that can capture meaningful change for progressive conditions.
- Risk-adjust or stratify by caseload complexity; report at the team level first, not individual ranking.
- Include a balancing measure (e.g., referrals declined, access wait times) to catch unintended consequences.
- Pilot before go-live, with a defined review and appeal process.
2. How would you implement and evaluate the revised approach, and what would you do about the declining referrals happening now?
Read the worked answer
Address the harm now, before the metric goes live:
- Raise the declining-referral pattern with administration as an unintended consequence and a patient-access problem, with the numbers (referrals declined, wait times) as evidence.
- Explicitly decouple the pilot from performance reviews until validity is established.
Implementation (6-month runway):
- Convene a small workgroup including front-line therapists; agree on the measure set, population rules, and data workflow.
- Build a data dictionary and test extraction on last year's 210 discharges to see how many would be excluded and whether the numbers are trustworthy.
- Train staff on the 'why' - that the goal is improvement in care, not surveillance.
- Run a 3-month pilot with team-level reporting; collect balancing measures (declined referrals, wait times, staff feedback).
Evaluation:
- Predefine success criteria: data completeness, whether the measure discriminates as expected, whether balancing measures worsen, and staff-reported burden.
- Use plan-do-study-act cycles; revise or abandon if the pilot shows gaming or access harm.
- Report results transparently to staff and administration, including what did not work.
Framing: the quality question is whether care improved, not whether individual therapists can be ranked. Keep the metric tied to that question.