Research, Measurement, and Program Evaluation
What you must know
Understand research designs - experimental (random assignment, causal inference), quasi-experimental, correlational, and single-subject - and threats to validity (history, maturation, testing, instrumentation, regression to the mean, selection, attrition). Know internal validity (did the manipulation cause the effect?) versus external validity (does it generalize?). Learn levels of measurement (nominal, ordinal, interval, ratio) and their permissible statistics. Understand reliability types (internal consistency, test-retest, inter-rater) and validity types (content, construct, criterion, face). Grasp sampling, effect size, statistical vs. clinical significance, Type I and Type II error, and basic program-evaluation models (needs assessment, process, outcome, cost). Know ethical research conduct: IRB review, informed consent, and protection of vulnerable populations.
Traps in questions
- Correlation presented as causation.
- Statistical significance equated with practical importance. Effect size answers the practical question.
- Confusing Type I (false positive) with Type II (false negative) error.
- Choosing a parametric statistic for ordinal data.
- Ignoring regression to the mean when interpreting extreme scores.
- Confusing internal with external validity.
Worked rule of thumb
When a stem describes a study and asks what limits its conclusions, first ask: Was there random assignment, a comparison group, and control of confounds? If not, causal language is unsupported. For measurement questions, apply the hierarchy - nominal (names), ordinal (ranks), interval (equal units, no true zero), ratio (true zero) - and choose statistics accordingly. Design determines what you may claim; effect size determines whether it matters.