Data Science: Data Lifecycle, Quality and Reproducibility
What the exam tests
This domain examines whether you can take a dataset from source to defensible analysis: framing the business question, assessing data quality, preparing data without leaking information, and making the work reproducible.
Core rules
- The data lifecycle follows CRISP-DM: business understanding, data understanding, data preparation, modelling, evaluation, deployment. Evaluation is not the end; monitoring and feedback belong in the lifecycle.
- Judge data quality on completeness, accuracy, consistency, timeliness, validity and uniqueness. Record the definition and provenance of every field.
- Separate training, validation and test data before any preprocessing that learns from data (imputation, scaling, target encoding). Otherwise information leaks from test to train and performance is overstated.
- Reproducibility requires versioned data and code, pinned environments, fixed random seeds and documented decisions.
Worked example
An insurer models 28-day lapse on 100,000 policies; 60% do not lapse. The analyst imputes missing income using the mean of the full dataset, then splits 70/30. Holdout AUC is 0.74. Repeating the process with the split first and computing the imputation mean on training data only gives 0.71. The first figure is optimistic because the imputation used holdout values. The correct workflow fixes the split, derives all transformations on training data, applies them to holdout, and reports 0.71.
Traps
- Reporting accuracy on a 60/40 imbalance as if it were informative; use AUC, lift or calibration by segment.
- Treating missingness as random when it may be informative, for example missing income for unemployed applicants.
- Using future information, such as a claim flag recorded after the observation date.
- Confusing a held-out test set with a validation set used for tuning.