Research and Program Evaluation
Designs and the questions they answer
Research items test your ability to name a design, spot a validity threat, and interpret a result.
Designs: experimental (random assignment plus a manipulated independent variable - the only design that supports causal claims), quasi-experimental (intact groups, interrupted time series), correlational (relationship only), single-subject (ABA, multiple baseline), and causal-comparative/ex post facto. Qualitative traditions: grounded theory (theory built by constant comparison), phenomenology (lived experience), ethnography (culture), case study (a bounded system), and consensual qualitative research.
Validity: internal threats include history, maturation, testing, instrumentation, regression to the mean, selection, and attrition (mortality). External threats include limited generalizability, reactivity, and demand characteristics.
Inference: null hypothesis testing, alpha, and p-values (the probability of the data given the null - not the probability the null is true). Type I error is a false positive; Type II is a false negative, and power is its complement. Effect sizes (Cohen's d: .2 small, .5 medium, .8 large) and confidence intervals matter more than p alone.
Program evaluation: needs assessment, formative versus summative, process versus outcome, logic models, and the CIPP model (context, input, process, product).
Ethics: IRB review, informed consent, voluntary participation, minimal risk, debriefing, confidentiality, Belmont principles (respect for persons, beneficence, justice), and disclosure of conflicts of interest.
Traps: claiming a correlational study proves causation; reading p = .04 as a large effect; confusing mortality with maturation; equating statistical significance with clinical importance.
Rule of thumb: before answering any methodology item, ask whether the IV was manipulated and participants randomized - that answer determines what causal language is permitted.