ICML cheat sheet

Bias-variance decomposition: expected squared error = bias^2 + variance + irreducible noise.

Bayes' theorem: P(A|B) = P(B|A)P(A)/P(B); posterior is proportional to likelihood times prior.

Gradient descent update: theta <- theta - eta * grad_theta L(theta); eta is the learning rate.

L2 regularization adds lambda*||w||^2 to the loss, shrinking weights; L1 adds lambda*||w||_1 and induces sparsity.

Central limit theorem: the sampling distribution of the mean approaches Normal(mu, sigma^2/n) as n grows.

Covariance matrix is symmetric positive semidefinite; its eigenvalues are nonnegative variances along eigenvectors.

A p-value is the probability of observing data at least as extreme as the sample, assuming the null hypothesis is true.

Lagrange multipliers solve constrained optimization by forming L = f + sum_i lambda_i g_i for equality constraints g_i = 0.

Standardization: z = (x - mu)/sigma; min-max scaling: x' = (x - min)/(max - min).

One-hot encoding creates k binary columns for a k-level categorical; drop one column to avoid perfect multicollinearity.

Target leakage occurs when a feature encodes information unavailable at prediction time or derived from the label.

Fit scalers and encoders on training data only, then apply the same parameters to validation and test sets.

Imputation choices: mean for symmetric numeric data, median for skewed data, mode for categoricals, or a missing-indicator flag.

Log or Box-Cox transforms reduce right skew and stabilize variance for positive-valued features.

Multicollinearity check: variance inflation factor VIF_j = 1/(1 - R_j^2); VIF above roughly 10 signals concern.

Class imbalance remedies include resampling, class weights, and threshold tuning rather than accuracy optimization.

Linear regression minimizes squared error; closed form w = (X^T X)^-1 X^T y when X^T X is invertible.

Logistic regression models p = sigmoid(w^T x) and is trained by minimizing cross-entropy loss.

Bias-variance tradeoff: high bias underfits (simple models), high variance overfits (complex models).

Decision trees split on impurity reduction, e.g., Gini or entropy information gain.

Random forests average many decorrelated trees via bootstrap sampling and random feature subsets, reducing variance.

Gradient boosting fits each new tree to the negative gradient (residuals) of the current ensemble's loss.

Support vector machines maximize the margin; the C parameter trades margin width against training-error tolerance.

k-nearest neighbors is non-parametric; predictions depend on the chosen k and the distance metric.

k-means minimizes within-cluster sum of squares; it requires specifying k and is sensitive to initialization.

The elbow method plots within-cluster SSE versus k; the silhouette score measures cohesion versus separation.

PCA projects onto orthogonal components ordered by explained variance, the top eigenvectors of the covariance matrix.

Hierarchical clustering uses linkage criteria (single, complete, average, Ward) and is read from a dendrogram.

DBSCAN groups dense regions using eps and minPts and labels sparse points as noise, discovering arbitrary shapes.

Gaussian mixture models fit a weighted sum of Gaussians via expectation-maximization, giving soft cluster assignments.

t-SNE and UMAP are nonlinear embedding methods for visualization; distances between far clusters are not reliable.

Anomaly detection flags points with low density or large reconstruction error relative to the learned normal distribution.

Precision = TP/(TP+FP); recall = TP/(TP+FN); F1 = 2precisionrecall/(precision+recall).

ROC-AUC is threshold-independent; PR-AUC is more informative under strong class imbalance.

k-fold cross-validation trains k models, each tested on a held-out fold; stratified folds preserve class proportions.

Nested cross-validation uses an inner loop for hyperparameter tuning and an outer loop for unbiased performance estimation.

Time-series data require forward-chaining or expanding-window splits, never random shuffling.

Regression metrics: MAE, RMSE (penalizes large errors more), and R^2 = 1 - SS_res/SS_tot.

A calibration curve compares predicted probabilities to observed frequencies; Platt scaling and isotonic regression recalibrate.

Data leakage across folds (e.g., scaling before splitting) inflates cross-validation estimates.

Backpropagation computes gradients via the chain rule; the loss gradient flows backward through each layer.

ReLU f(x)=max(0,x) mitigates vanishing gradients; sigmoid and tanh saturate at extremes.

Batch normalization normalizes layer activations per mini-batch, stabilizing and accelerating training.

Dropout randomly zeroes activations during training as a regularizer; it is disabled at inference.

Adam combines momentum with adaptive per-parameter learning rates using estimates of first and second moments.

Convolutional layers share weights across spatial positions, giving translation equivariance and parameter efficiency.

Attention computes softmax(QK^T/sqrt(d_k))V, weighting tokens by query-key similarity.

Vanishing/exploding gradients are countered by residual connections, gradient clipping, and careful initialization.

Training-serving skew arises when feature computation differs between training pipelines and live inference.

A feature store provides consistent offline (training) and online (serving) feature computation.

Model versioning pairs each artifact with code, data, and configuration hashes for reproducibility.

Shadow deployment sends live traffic to a new model without affecting users; canary release routes a small traffic fraction.

Monitoring covers data drift, concept drift, prediction distributions, latency, and error rates.

Batch inference precomputes predictions; online inference serves per-request predictions with latency constraints.

Rollback requires retaining the previous model artifact and its configuration until the new version is validated.

Pipelines should be idempotent and reproducible, with automated tests on data schemas and model outputs.

Demographic parity requires equal positive-prediction rates across groups; equalized odds requires equal TPR and FPR.

Disparate impact ratio compares selection rates between groups; the four-fifths rule flags ratios below 0.8.

Fairness metrics often conflict; satisfying multiple group-fairness criteria simultaneously is generally impossible.

Explainability tools include SHAP (Shapley-value attributions) and LIME (local surrogate models).

Model cards and datasheets document intended use, training data, performance, and limitations.

GDPR-style rights include explanation, access, and erasure, affecting how models and data must be retained.

Human-in-the-loop review is required for high-stakes automated decisions in many regulatory frameworks.

Governance controls include audit logs, bias testing before release, and approval gates for model changes.