Measurement, Instrumentation, and Data Collection
Measurement fundamentals
Candidates must understand reliability and validity as separate properties. Reliability concerns consistency and includes inter-rater, test-retest, and internal consistency. Validity concerns whether an instrument measures what it intends and includes content, construct, criterion, and face validity. Know that a measure can be reliable without being valid, but cannot be valid without being reliable.
Scales and instruments
Understand the levels of measurement (nominal, ordinal, interval, ratio) and how they constrain permissible statistics. Learn the meaning of sensitivity, specificity, positive and negative predictive value, and how prevalence affects predictive values. Know that a diagnostic test's operating characteristics depend on the threshold chosen.
Data collection practice
Know the elements of a sound data collection plan: standardized procedures, trained assessors, pilot-tested forms, and defined handling of missing values. Understand the role of calibration and quality control in reducing measurement error. Distinguish random from systematic measurement error; the former affects precision, the latter biases estimates.
Traps
- Confusing predictive value with sensitivity; predictive value depends on prevalence.
- Treating an ordinal scale as interval and computing means uncritically.
- Assuming high internal consistency proves validity.
- Overlooking that a change in instrument version or assessor can introduce systematic error mid-study.
Worked rule of thumb
Before trusting a number, ask: was it measured with a documented, reliable instrument, by a trained assessor, under a standardized protocol, with missing data handled by a pre-specified rule? If any answer is no, treat the measurement as a potential source of bias rather than noise.