Data Preparation and Data Quality
What this domain tests
This domain tests your ability to profile, clean, transform, and document data without introducing bias or leakage. The exam expects you to recognize data-quality dimensions, handle missing values and outliers deliberately, and keep preprocessing inside the correct validation boundary.
Core rules
- Profile before transforming. Check completeness, accuracy, consistency, timeliness, uniqueness, and validity. Record row counts, distributions, duplicates, and key relationships.
- Diagnose missingness. Distinguish MCAR, MAR, and MNAR. Mean imputation is acceptable only under strong assumptions and rarely for MNAR. Prefer domain rules, multiple imputation, or explicit missing indicators when justified.
- Treat outliers by cause, not by reflex. A data-entry error, a legitimate extreme, and a separate population require different actions.
- Fit transformations on training data only. Scaling, imputation, encoding, and feature selection must be learned from the training fold and applied to validation and test data.
- Preserve lineage and versioning. Document source tables, filters, joins, transformations, and dataset versions.
- Check sampling and join bias. Non-random missingness, survivorship bias, and many-to-many joins can distort results before modeling.
Worked example
A customer dataset has 12% missing income, duplicate account rows, and income outliers. First, deduplicate on account ID using a documented rule and verify row counts. Profile missingness by segment: if high-value customers are more likely to omit income, the data are not missing completely at random, so mean imputation would bias the analysis. Use a missing indicator plus a model-based imputation, or exclude income if it is not essential. Cap or winsorize extreme values only after confirming they are plausible. Split into training and test sets before fitting the imputer and scaler; fit both on training only. Report the missingness rate and imputation method in the data dictionary.
Traps that cost marks
- Imputing or scaling before the train/test split, which leaks test information.
- Dropping all rows with missing values when the missingness is informative.
- Removing outliers without checking whether they are real and important.
- Joining tables without validating cardinality, creating duplicate rows.
- Forgetting that categorical encoding must be fit on training categories only.
- Treating a cleaned dataset as reproducible without version control.