Predictive Analytics (PA) Exam
Predictive analytics workflow
The Predictive Analytics exam tests the full modeling pipeline: problem framing, data cleaning, feature engineering, model selection, validation, interpretation, and communication. Know regression, GLMs, regularization, trees, random forests, boosting, and neural networks at a conceptual level. Understand bias-variance tradeoff, cross-validation, train-validation-test splits, leakage, and performance metrics such as RMSE, MAE, AUC, and deviance. A model is useful only if it generalizes beyond the training data.
Practical traps
Data leakage occurs when information unavailable at prediction time enters training. Overfitting shows excellent training performance but poor validation performance. Accuracy can mislead on imbalanced data; use precision, recall, or AUC as appropriate. Do not tune on the test set. For interpretability, remember that variable importance is not causation and can be unstable with correlated predictors. Check missing-value handling and whether categories were encoded consistently.
Worked rule
With 10-fold cross-validation, fit the model 10 times, each time holding out one fold, and average the validation metric. Use the average to compare models, then refit the chosen model on all training data before final testing. The trap is choosing the model with the best single fold or evaluating on the test set repeatedly. A useful rule: cross-validation guides choices; the untouched test set estimates final performance.