Research Methods and Statistics
Research stems test design judgment and statistical interpretation. They often present a finding and ask what can and cannot be concluded: whether a threat explains the result, whether the test chosen was appropriate, or whether a claim about mechanism is supported.
Validity threats and design
- Internal validity threats: history, maturation, testing, instrumentation, regression to the mean, attrition, selection. Random assignment addresses selection and distributes some threats, but not attrition or instrumentation bias.
- External validity concerns generalizability across populations, settings, and time; analogue studies trade realism for control.
- Random selection supports generalizability; random assignment supports causal inference.
- Single-case designs (ABAB reversal, multiple baseline) can establish experimental control without a control group.
Choosing a statistic
- Nominal data: chi-square. Ordinal data: nonparametric tests such as Mann-Whitney, Wilcoxon, or Kruskal-Wallis.
- Two groups with a continuous outcome: t-test; three or more: ANOVA; multiple dependent variables: MANOVA; adjusting for a covariate: ANCOVA.
- Type I error is alpha (rejecting a true null); Type II is beta (failing to reject a false null); power equals 1 minus beta and rises with sample size, effect size, and alpha.
- Effect size: Cohen's d around .2 is small, .5 medium, .8 large. Statistical significance is not clinical significance.
Moderation and mediation
- A moderator specifies for whom or when an effect holds and appears as an interaction.
- A mediator specifies how or why an effect operates (X to M to Y) and is tested through indirect effects.
Worked example
A randomized trial finds an exercise program reduces depressive symptoms only among participants with high baseline inflammation, and a separate analysis shows the effect is partly explained by improved sleep. Reasoning: baseline inflammation is a moderator because the treatment effect differs across its levels, so the interaction must be reported rather than the main effect alone. Sleep is a mediator because it lies on the causal pathway. The moderator tells a clinic whom to prioritize; the mediator suggests a mechanism but does not by itself prove causation.
Traps
- Reading a p value as the probability that the null hypothesis is true.
- Assuming a large sample corrects measurement error or systematic bias.
- Confusing statistical significance with clinical importance.
- Treating a cross-sectional path model as proof of causal mediation.
- Attributing improvement to treatment when regression to the mean or history accounts for it.