Data Mining
Data Mining
Data mining is the process of finding patterns in large data sets and turning them into models. The standard lifecycle is CRISP-DM: business understanding, data understanding, data preparation, modeling, evaluation, and deployment. Expect questions that ask which phase a task belongs to - cleaning and feature engineering sit in data preparation, not modeling.
Know the task types. Classification predicts a categorical label; regression predicts a continuous value; clustering groups records without labels (k-means, hierarchical); association rule mining finds co-occurring items (market basket); anomaly detection flags rare events; time-series forecasting predicts future values from ordered history.
Then know evaluation. Split data into training and test sets; use cross-validation when data is limited. Overfitting means great training performance and poor generalization - often from too many features or too little data; underfitting means the model is too simple. For classification, build a confusion matrix and compute accuracy, precision, recall, and F1. Accuracy misleads on imbalanced data, so prefer precision/recall or AUC. For regression, use MAE, RMSE, or R-squared. For clustering, use silhouette score or the elbow method.
Traps: confusing unsupervised clustering with classification; treating association-rule confidence as lift; ignoring data leakage, such as using target-derived fields as features; and assuming a high R-squared proves causation.
Worked example: a grocery chain finds {bread, butter} with support 0.05, confidence 0.60, and lift 1.8. Lift greater than 1 means the items co-occur more than chance, so the rule is worth acting on; confidence alone can be high for popular items.