Evaluating Analytical Approaches and Models
What this domain tests
This domain tests whether you can choose an approach that fits the problem, validate it honestly, and select a model using metrics tied to the decision. Expect questions on baselines, cross-validation, overfitting, class imbalance, and metric choice.
Core rules
- Match the approach to the task. Use regression for continuous outcomes, classification for discrete outcomes, clustering for unsupervised segmentation, and time-series methods for ordered dependence.
- Always establish a baseline. Compare against a simple rule, current process, or null model.
- Choose validation that mirrors deployment. Use stratified k-fold for imbalanced classification, grouped splits when entities repeat, and time-ordered splits for temporal data.
- Select metrics by cost and prevalence. Accuracy is misleading under imbalance. Use precision, recall, F1, PR-AUC, or cost-weighted metrics; use RMSE, MAE, and R² for regression with attention to scale.
- Tune inside training folds. Hyperparameter search on the test set overfits the evaluation. Use nested cross-validation when you need an unbiased estimate.
- Balance performance with interpretability, stability, latency, and maintainability.
Worked example
A fraud team has 0.5% positive cases. A model with 99.5% accuracy may simply predict “not fraud.” The team needs to catch fraud while limiting false alerts. Start with a rule-based baseline. Use stratified cross-validation. Evaluate PR-AUC and recall at a fixed precision that operations can handle, not accuracy alone. Tune the decision threshold against the cost of missed fraud versus manual review. Compare logistic regression and gradient boosting; if performance is close, prefer the simpler, more stable model. Report the chosen threshold and the estimated review volume. This reasoning connects model selection to the operational decision.
Traps that cost marks
- Reporting accuracy on a heavily imbalanced target.
- Tuning hyperparameters on the test set.
- Using random cross-validation on time-series or grouped data.
- Ignoring a baseline that a simple rule already achieves.
- Choosing a complex model without a measurable gain.
- Omitting the decision threshold when precision and recall trade off.