Machine Learning Fundamentals
This domain tests the discipline around a model: honest evaluation, the right metric for the problem, and knowing why a method fails.
Setup and the bias-variance trade-off
Split into train, validation, and test; cross-validate on the training portion only and touch the test set once. High bias underfits, high variance overfits. Counter overfitting with regularisation (L1 sparsifies coefficients, L2 shrinks them), early stopping, pruning, or more data. Scaling, imputation, and target encoding must be fitted inside the training fold, otherwise information leaks.
Metrics
From the confusion matrix: precision = TP/(TP+FP), recall = TP/(TP+FN), F1 is their harmonic mean. Accuracy misleads under class imbalance; ROC-AUC is threshold-free but optimistic when positives are rare, where PR-AUC is more informative. For regression: MAE, RMSE (penalises large errors), and R-squared, which can be negative out of sample.
Algorithms
- Linear and logistic regression: interpretable, need scaling when regularised.
- k-NN: sensitive to scaling and to the choice of k.
- Trees: axis-aligned splits, low bias, high variance; random forests bag them, gradient boosting fits trees sequentially to residuals.
- k-means: choose k, scale features, initialisation matters. PCA: standardise first, components are orthogonal, inspect variance explained.
Worked example
A classifier yields TP = 40, FP = 60, FN = 10, TN = 890 out of 1,000. Precision = 40/100 = 0.40; recall = 40/50 = 0.80; F1 = 2(0.40)(0.80)/1.20 = 0.53; accuracy = 930/1,000 = 0.93. Accuracy looks excellent while three of every five flagged cases are wrong, which is why the metric must match the cost of errors.
Traps
Fitting transformers before the split; tuning on the test set; selecting features using the full dataset; reporting accuracy alone; and reading feature importance as causal evidence.