Supervised Learning
What to know
Expect coverage of the bias-variance tradeoff, regularization (L1 sparsity, L2 shrinkage, elastic net), linear and logistic regression (link functions, odds, log-loss), decision trees and their instability, ensembles (bagging reduces variance, boosting reduces bias), random forests, gradient boosting, SVMs (margins, kernels, C), k-NN (curse of dimensionality, distance choice), and naive Bayes (conditional independence assumption). Understand loss functions and how model capacity interacts with data size.
Traps
- Claiming boosting reduces variance primarily - it mainly reduces bias; bagging reduces variance.
- Forgetting that L1 performs feature selection while L2 shrinks coefficients smoothly.
- Interpreting logistic regression coefficients as additive changes in probability rather than in log-odds.
- Assuming deeper trees always generalize better; they overfit.
- Mixing up precision/recall tradeoffs controlled by thresholds and class weights.
Rule of thumb
Start with a regularized linear baseline; move to gradient boosting for tabular data; reserve deep nets for unstructured data or very large samples.
Worked example
A logistic model reports coefficient 0.7 for a feature. The odds multiply by e^0.7 ≈ 2.01 per unit increase, holding others fixed - a doubling of odds, not a doubling of probability. Multiply by e^β, never add β to p.