Data Preparation, Feature Engineering, and Model Evaluation
What this domain tests
How you turn raw, messy data into leakage-free features and how you choose an evaluation protocol that will still hold in production. Most failed models in practice fail here rather than in the algorithm.
Core rules
- Fit every transformation - imputation, scaling, encoding, feature selection - inside the training fold only. Wrap them in a pipeline so cross-validation refits preprocessing per fold.
- Handle missingness deliberately: median imputation compresses variance, so add a missing-indicator column; data missing not at random needs a domain explanation, not a fill value.
- Encoding: one-hot for low cardinality, out-of-fold target encoding with smoothing for high cardinality, hashing when cardinality explodes.
- Scaling matters for distance- and gradient-based models; tree ensembles are largely scale-invariant.
- Imbalance: use stratified splits, class weights, or resampling, and report precision, recall, F1 and PR-AUC.
- Split by group or by time when rows are not independent; random k-fold leaks future information into the past.
Worked example
Fraud detection with 1% positives. A model that always predicts \u201cnot fraud\u201d reaches 99% accuracy with zero fraud recall. Reasoning: accuracy measures the majority class the model already gets for free. Rank models with PR-AUC, then set the decision threshold against the operations team's review capacity, because precision and recall trade off directly at that threshold.
Traps that cost marks
Scaling or imputing before the split. Computing target encodings on the full dataset. Tuning the threshold on the test set. Trusting ROC-AUC under extreme imbalance, where it stays flattering while precision collapses. Dropping all rows with missing values when the missingness itself is predictive.