Data - Types, Collection, Cleaning and Descriptive Statistics
Data work begins with types, collection and cleaning because descriptive statistics are only as good as the underlying observations.
Core concepts
- Structured data fit rows and columns; unstructured data include text, images and audio.
- Measurement scales: nominal (labels), ordinal (rank), interval (equal differences, arbitrary zero) and ratio (true zero).
- Primary data are collected for the current question; secondary data already exist.
- Cleaning checks missing values, duplicates, inconsistent formats, impossible values and outliers. A data dictionary and reproducible cleaning log prevent silent changes.
- Descriptive statistics: mean, median, mode, range, variance, standard deviation, percentiles and skewness.
Worked example
For observations 4, 6, 6, 8, 10, the mean is 34/5 = 6.8 and the median and mode are both 6. Deviations from the mean are −2.8, −0.8, −0.8, 1.2 and 3.2; squared deviations sum to 20.8. Sample variance = 20.8/4 = 5.2 and sample standard deviation = √5.2 = 2.28. The mean exceeds the median, a sign of right skew.
Traps
Do not average ordinal ranks as if they were ratio data; do not delete missing values without checking whether they are missing at random; outliers may be errors or genuine; and the mean is sensitive to skew while the median is robust.