Data Science and Analytics
This domain tests the analytics lifecycle from problem framing through deployment. CDP candidates should know how to prepare data without leakage, select methods, evaluate models with appropriate metrics, and monitor results after release.
Core rules and concepts
- Lifecycle: CRISP-DM frames business understanding, data understanding, preparation, modeling, evaluation, and deployment. Start with a baseline and a decision-focused success metric.
- Problem type: supervised learning uses labeled data for classification or regression; unsupervised learning includes clustering and association; time series requires temporal validation.
- Data preparation: fit imputation, encoding, and scaling on training data only, then apply to validation and test data. Leakage occurs when information unavailable at prediction time enters training.
- Evaluation: a confusion matrix yields precision, recall, and F1. ROC-AUC summarizes ranking across thresholds; precision-recall curves are more informative for rare events. Regression uses MAE, RMSE, and R² with attention to error scale.
- Generalization: overfitting produces low training error and poor validation performance. Cross-validation, regularization, and feature selection reduce variance.
- Deployment: monitor drift, data quality, and business outcomes; preserve reproducibility through versioned data, code, and parameters.
Worked example
Fraud detection has 1% positives. A model predicting “not fraud” scores 99% accuracy but catches nothing. Precision and recall reveal the tradeoff: lowering the threshold raises recall but may overwhelm investigators with false positives. The team uses stratified cross-validation, reports a precision-recall curve, and chooses a threshold from investigation cost and loss avoided. The test set is untouched until final evaluation.
Traps that cost marks
Reporting accuracy on imbalanced data; tuning on the test set; confusing correlation with causation; dropping missing values without checking the mechanism; leaking target-derived features; and deploying without drift or performance monitoring.