Measurement and Methodology
Measurement and Methodology tests whether you can judge a study: are the measures reliable and valid, is the design causal, is the statistic appropriate, and does the conclusion follow? Expect items that present a design or a number and ask for the flaw, the correct test, or the correct interpretation.
Reliability and validity
- Reliability: test–retest, parallel forms, split-half (corrected with Spearman–Brown), inter-rater. Reliability is necessary but not sufficient for validity and caps how valid a measure can be.
- Validity: face (looks appropriate), content (covers the domain), criterion (concurrent and predictive), construct (convergent and discriminant).
- A scale can be perfectly reliable and wrong: a ruler that reads two inches too long is consistent and invalid.
Scales and statistics
- Nominal (categories), ordinal (rank with unequal intervals), interval (equal units, no true zero), ratio (true zero). Celsius is interval; height is ratio.
- The mean is pulled by skew; the median resists it. Standard deviation measures spread; z-scores express distance in SD units.
- Correlation runs from −1 to +1 and conveys direction and strength, never causation. Restriction of range attenuates r.
- Type I error (alpha): rejecting a true null. Type II error (beta): failing to reject a false null. Power rises with effect size, sample size, and alpha.
Design
- Random assignment to condition supports causal inference; random sampling supports generalization. Confusing the two is a classic error.
- Between-subjects designs compare different people; within-subjects designs control individual differences but invite order effects, countered by counterbalancing.
- Threats to internal validity: confounds, demand characteristics, placebo effects, experimenter bias, maturation, attrition, regression to the mean.
- A mediator explains how an effect occurs; a moderator changes its strength or direction.
Worked example
A new anxiety scale correlates .92 with itself across two weeks but only .10 with clinicians' ratings. Reasoning: the high test–retest coefficient establishes reliability, while the low correlation with an established criterion shows poor concurrent validity. The correct conclusion is not "it measures anxiety reliably, so it is useful" - it is consistently measuring something the clinicians do not see. Also note that the reliability coefficient sets a ceiling on the validity coefficient that could be obtained.
Traps
- Calling a study experimental when assignment was not random; it is quasi-experimental.
- Thinking a lower alpha means fewer Type I errors, when alpha is the Type I error rate you choose.
- Treating an ordinal scale as interval, for example by averaging ranks.