Program Review When Outcomes Data Are Missing
A medical technology program undergoes periodic accreditation review. The self-study claims strong graduate outcomes but provides: graduation rates for the last 3 years (88%, 91%, 90%), a licensure first-attempt pass rate cited as 'above national average' with no numbers, and employer satisfaction described as 'very positive' from an informal survey of 9 employers. Graduate employment data are missing for 40% of the last cohort. The standards require evidence of graduate outcomes, including licensure performance, employment, and program evaluation used for improvement. The review team must decide whether the evidence is sufficient.
1. Evaluate the sufficiency of the outcomes evidence against the standard.
Read the worked answer
Assessment by element:
- Graduation rates: Adequate - three years of data with a consistent trend.
- Licensure pass rate: Insufficient. 'Above national average' without numbers cannot be verified. The standard requires actual first-attempt pass rates and the comparison source. A claim without data is not evidence.
- Employer satisfaction: Weak. A survey of 9 employers with no response rate, instrument, or results is anecdotal. It may support, but cannot carry, the standard.
- Employment data: Deficient. Missing 40% of the last cohort is a large non-response; the remaining 60% may be biased (e.g., employed graduates more likely to respond).
- Program evaluation: No evidence that outcomes data are used for improvement.
Conclusion: The evidence is partially sufficient. The program meets the graduation element but not licensure, employment, or evaluation-for-improvement. This is a substantive gap, not a formatting issue.
2. Recommend what the program must submit and how the review team should treat the missing employment data.
Read the worked answer
Required submissions:
- Licensure: Actual first-attempt pass rates for each of the last 3 years, the number of candidates, and the national comparator with its source and year.
- Employment: A defined response rate and method; report the 40% missing as a limitation. If non-response is high, conduct a targeted follow-up (e.g., phone or registry match) to reduce bias.
- Employer survey: Instrument, response rate, and item-level results.
- Evaluation loop: Documentation showing how outcomes data led to a specific change (e.g., curriculum revision) and its effect.
Treatment of missing data: Do not impute or ignore. Report employment outcomes with the response rate and a sensitivity statement. If the missing 40% cannot be characterized, treat the employment element as not demonstrated and require improvement.
Decision: Defer or condition the decision on submission of verifiable licensure and employment data within a set period.
3. The program says the missing data are due to graduates not responding. Propose a defensible improvement requirement.
Read the worked answer
Requirement (specific and measurable):
- Establish an outcomes tracking system with a named responsible person and a schedule (e.g., contact at graduation, 6 months, and 12 months).
- Set a minimum response-rate target (e.g., 70%) and require reporting of the rate each cycle.
- Use multiple sources - graduate survey, employer survey, and licensure registry - to cross-check.
- Report limitations when the target is not met and describe the follow-up attempted.
- Close the loop: Show at least one documented use of outcomes data for program improvement per cycle.
Monitoring: Require a progress report at 12 months with the response rate and the first improvement action. This converts a data gap into a verifiable process requirement, which is what accreditation monitoring is for.