Statistical Computing and Data Handling
This domain tests the practical craft of getting data into an analysis without corrupting it: data types, cleaning, missing values, reproducibility, and the numerical and indexing habits that cause silent errors.
Data structures and cleaning
- Know your types: numeric, integer, character, factor or categorical, and date-time. Sorting a factor by its labels rather than its defined levels is a classic mistake.
- Tidy layout means one row per observation, one column per variable and one table per unit of analysis.
- Missing values need a recognised missing code, not a sentinel such as minus 999 or 0, which will silently enter means and totals.
- Before analysis, check for duplicate records, impossible ranges, inconsistent units, inconsistent capitalisation and stray whitespace.
Reproducibility
- Set the random seed before any simulation, bootstrap or random split so the result can be repeated exactly.
- Keep scripts rather than mouse clicks; record package and software versions; keep raw data immutable and derive the cleaned file by script so the transformation can be audited.
Numerical and indexing pitfalls
- R indexes from 1 and Python from 0; an off-by-one error quietly shifts which row is analysed.
- Floating point arithmetic is not exact: 0.1 plus 0.2 is not exactly 0.3, so compare with a tolerance or an equality-checking function rather than with a strict comparison.
- Vectorised operations replace loops: they are faster and less prone to indexing mistakes.
- Rounding for display should happen at the end, not in the middle of a calculation chain.
Worked example
Suppose a vector holds 12, missing, 15 and 9. The mean computed with missing values removed is 36/3 = 12, whereas the default mean returns a missing value because one element is unknown. Testing equality against the missing code returns missing for every element, while the missing-value test returns true, false, false, false and works correctly. Had the missing entry been entered as 0, the mean would have been 9, a 25% understatement caused purely by data handling.
Traps
- Dropping incomplete rows without checking whether missingness is related to the outcome.
- Comparing floating point results with strict equality.
- Overwriting the raw data file so the cleaning step cannot be reproduced.
- Setting the seed after the random draw has already happened.