Research and Program Evaluation
What this domain tests
Research designs, threats to validity, basic statistics, qualitative methods, and program evaluation, plus the ethics of human-subjects research.
Core content
- Designs: experimental (random assignment, control), quasi-experimental, correlational, survey, and single-subject designs (ABAB reversal, multiple baseline).
- Threats to internal validity: history, maturation, testing effects, instrumentation, regression to the mean, selection, and attrition/mortality. Threats to external validity concern generalizability, including reactive effects of being observed.
- Statistics: central tendency (mean, median, mode), variability (range, variance, standard deviation), the normal curve (about 68-95-99.7 percent within 1, 2, and 3 SD). Type I error = alpha (false positive); Type II = beta (false negative); power = 1 - beta. Effect size (Cohen's d) answers whether a difference matters.
- Analyses: t-test for two means, ANOVA for three or more (factorial designs test interactions), chi-square for categorical data, Pearson r and regression for association and prediction.
- Qualitative: grounded theory, phenomenology, ethnography, case study, consensual qualitative research. Lincoln and Guba's trustworthiness criteria: credibility, transferability, dependability, confirmability.
- Program evaluation: needs assessment; formative (improvement) vs summative (judgment); the CIPP model (context, input, process, product). IRB review, informed consent, and debriefing after deception.
Worked example
A school asks whether a new counseling curriculum reduces discipline referrals. Reasoning: use a comparison school and pre/post data (quasi-experimental, since random assignment is impractical); watch history and maturation; analyze with a t-test or ANOVA; report an effect size, not just p; feed results back formatively.
Traps that cost marks
- Correlation does not establish causation.
- Statistical significance is not practical significance.
- A p value is not the probability that the null hypothesis is true.
- Attrition is not the same threat as maturation.