Evaluation and Research
What you must know
Know the types by purpose: formative (improve before/during), process (was it delivered as planned - reach, dose, fidelity), impact (changes in knowledge, attitudes, skills, behavior), outcome (health status, quality of life), and summative (overall judgment against goals). Match design to question: randomized controlled trial, quasi-experimental (nonequivalent comparison group, interrupted time series), pre-experimental (one-group pre/post), cross-sectional, cohort, case-control. Recognize threats to internal validity - history, maturation, testing, instrumentation, selection, attrition, regression to the mean - and distinguish internal from external validity. Understand reliability, measurement validity, indicator selection, mixed methods, and basic analysis: descriptive versus inferential statistics, the p-value as the probability of observed data under the null (not the probability the null is true), confidence intervals, and practical versus statistical significance. Know IRB review categories and cost analyses (cost-effectiveness, cost-benefit, cost-utility). Frameworks such as RE-AIM and utilization-focused or empowerment evaluation are fair game.
Traps questions set
- Calling a behavior change an outcome when it is an impact indicator.
- Confusing process evaluation with formative evaluation.
- Misreading p < .05 as a large or important effect.
- Confusing internal validity (did the program cause the change) with external validity (does it generalize).
Rule of thumb
Impact = risk factors and behaviors. Outcome = health status and quality of life. Place any indicator accordingly before answering.
Worked example: A smoking program reports that 40% of participants quit (impact) and that clinic asthma visits fell 15% (outcome). Both are results, but only the second is a health-status outcome.