Research and Program Evaluation
What this domain covers
Research and Program Evaluation tests designs, validity, sampling, statistics, research ethics, and the logic of program evaluation.
Core rules
- Designs: experimental (random assignment, manipulation, control), quasi-experimental (no random assignment), correlational, ex post facto, single-case, and qualitative approaches (phenomenology, grounded theory, ethnography, case study).
- Internal validity threats: history, maturation, testing, instrumentation, regression to the mean, selection, attrition, and diffusion. External validity concerns generalization.
- Random selection affects generalizability; random assignment supports causal inference.
- Belmont Report principles: respect for persons, beneficence, justice. IRB approval, informed consent, and debriefing after deception are required.
- Probability sampling: simple random, systematic, stratified, cluster. Nonprobability: convenience, purposive, quota, snowball.
- Statistics: mean, median, mode, range, standard deviation; normal curve percentages; null hypothesis testing; Type I error (alpha, false positive) versus Type II error (beta, false negative); power equals 1 minus beta; p-value; effect size such as Cohen's d. Tests: t-test, ANOVA, chi-square, Pearson r.
- Program evaluation: needs assessment, formative (process) and summative (outcome) evaluation; the CIPP model (context, input, process, product).
Worked example
A study reports p = .04 with alpha at .05 and a small effect size. Reasoning: the null hypothesis is rejected, but there is a 4% chance of a Type I error if the null is true, and the small effect size suggests limited practical significance. A counselor reading this should check the sample, the design, and the confidence interval before changing practice.
Traps
Confusing Type I with Type II error; concluding that correlation demonstrates causation; reading p as the probability the null is true; confusing random assignment with random selection; generalizing from a convenience sample.