Research and Program Evaluation
Research and Program Evaluation tests design literacy, statistics, and evaluation logic: can you spot the threat to validity, name the error, and judge whether a program worked?
Core
- Designs: experimental (random assignment), quasi-experimental (nonequivalent groups, time series), correlational, single-subject (ABAB, multiple baseline), qualitative.
- Internal validity threats: history, maturation, testing, instrumentation, regression to the mean, selection, attrition, diffusion of treatment. External threats: generalizability, reactivity.
- Inferential statistics: null hypothesis, alpha (Type I - rejecting a true null), beta (Type II - failing to reject a false null), power, p-value, effect size (Cohen's d ≈ .2/.5/.8).
- Qualitative trustworthiness: credibility, transferability, dependability, confirmability.
- Program evaluation: needs assessment, formative vs. summative, CIPP (context, input, process, product), logic models.
Worked example
An agency compares a new CBT group with a waitlist. Participants self-selected the group and scored higher on depression at baseline. Post-test differences favor CBT. Reasoning: the design is quasi-experimental; baseline differences create a selection threat, and regression to the mean may inflate improvement. Naming the threat is not enough - the fix is randomization, matching, or ANCOVA controlling baseline.
Traps
- Saying p is the probability the null is true.
- Equating statistical significance with practical importance.
- Swapping Type I and Type II errors.
- Treating correlation as causation.
- Calling a pretest-posttest single-group design experimental.