Experimental Design and Sampling
This domain covers how data are generated: the principles that make an experiment informative and the schemes that make a sample representative of the population you want to describe.
Design principles
- Randomisation of treatments protects against confounding by unmeasured variables.
- Replication supplies an internal estimate of variability, without which no effect can be tested.
- Blocking removes known nuisance variation by grouping similar units and randomising within blocks.
- Factorial designs estimate main effects and interactions; a completely randomised design has one factor, a randomised block design adds blocks.
- Control groups and blinding reduce placebo effects and assessment bias.
Sampling methods
- Simple random sampling gives every unit an equal, known chance and needs a complete list, the sampling frame.
- Stratified sampling divides the population into internally homogeneous strata and samples every stratum; it reduces variance when stratum means differ.
- Cluster sampling selects whole clusters, which is cheaper but less precise because units within a cluster resemble one another.
- Systematic sampling takes every k-th unit; it fails if the list has a periodic pattern that matches k.
Sources of error
- Sampling error shrinks as n grows. Non-sampling error, such as coverage, non-response, measurement and processing error, does not necessarily shrink.
- Undercoverage and voluntary response create bias, which larger samples cannot fix.
Worked example
A population has stratum A with 1,500 units and mean 20, and stratum B with 2,500 units and mean 32. Proportional allocation of a sample of 200 gives 75 from A and 125 from B. The stratified estimate of the mean is 0.375 times 20 plus 0.625 times 32 = 7.5 + 20 = 27.5, not the unweighted 26 that a simple average of the two stratum means would give.
Traps
- Confusing stratified sampling, where strata differ and all are sampled, with cluster sampling, where clusters are similar and only some are sampled.
- Reporting sampling error as though it were the only error present.
- Allocating a stratified sample equally when proportional allocation was intended.
- Ignoring omissions in the sampling frame that exclude part of the target population.