ICML key terms

The tension between a model that is too simple and misses real patterns (high bias) and one that is too flexible and fits noise (high variance). Total expected error decomposes into bias squared, variance, and irreducible noise.

A method of estimating parameters by choosing values that make the observed data most probable under the assumed model. For many models it coincides with minimizing a negative log-likelihood loss.

A rule relating conditional probabilities: the posterior probability of a hypothesis given evidence equals the likelihood times the prior, divided by the marginal probability of the evidence.

States that the sampling distribution of a sample mean approaches a normal distribution as sample size grows, regardless of the underlying population shape, provided observations are independent and variance is finite.

A framework for deciding whether observed data are compatible with a null hypothesis, using a test statistic and a threshold on its probability under the null.

The probability, assuming the null hypothesis is true, of observing a test statistic at least as extreme as the one computed. It is not the probability that the null hypothesis is true.

A range computed from data that, under repeated sampling, would contain the true parameter value a specified fraction of the time. Width reflects uncertainty and shrinks with more data.

Adding a penalty on model complexity, such as L1 or L2 norms on weights, to the training objective in order to reduce variance and improve generalization.

For a square matrix, an eigenvector is a nonzero vector whose direction is unchanged when the matrix is applied, and its eigenvalue is the scalar by which it is scaled. Central to PCA and stability analysis.

A factorization of any matrix into rotation, scaling, and rotation components. It underlies dimensionality reduction, low-rank approximation, and pseudoinverse computation.

Filling in absent feature values using statistics such as the mean, a model-based prediction, or iterative methods. The mechanism of missingness affects whether simple imputation introduces bias.

Transforming numeric features to comparable ranges, for example by standardization or min-max normalization. Required for distance-based and gradient-based methods and must be fit on training data only.

Representing a categorical variable as binary indicator columns, one per category. Avoids implying order but can create high dimensionality and collinearity if all columns are kept.

Replacing a categorical level with a statistic of the target computed from training rows, such as the mean outcome. Powerful for high-cardinality features but prone to leakage unless computed out-of-fold.

When information unavailable at prediction time influences training, such as statistics computed over the full dataset or future rows. It inflates validation scores and produces models that fail in production.

A setting where one class greatly outnumbers another, causing accuracy to be misleading and models to favor the majority class. Addressed with resampling, class weights, or threshold tuning.

A resampling procedure that splits data into folds, training on some and validating on others, then averaging results. Gives a more stable estimate of generalization than a single split.

Searching over configuration choices not learned from gradients, such as tree depth or learning rate, using validation performance. Search must be nested or separate from final evaluation data.

When a model captures noise specific to the training sample, achieving low training error but poor performance on new data. Detected by a widening gap between training and validation metrics.

Precision is the fraction of predicted positives that are truly positive; recall is the fraction of actual positives that were found. They trade off against each other as the decision threshold moves.

The area under the receiver operating characteristic curve, equal to the probability that a randomly chosen positive is scored above a randomly chosen negative. Threshold-independent but insensitive to calibration.

A table of counts of predicted versus actual classes, from which accuracy, precision, recall, and related metrics are derived. It exposes which classes are being confused.

The agreement between predicted probabilities and observed frequencies. A well-calibrated model that outputs 0.8 should be correct about 80 percent of the time on such predictions.

A supervised model predicting a continuous target as a weighted sum of input features plus an intercept, typically fit by minimizing squared error. Coefficients are interpretable but assume linearity.

A linear classifier that maps a weighted sum of features through a sigmoid to produce class probabilities, trained by minimizing cross-entropy. Despite the name, it is a classification method.

A model that recursively splits feature space using threshold rules chosen to reduce impurity, producing a tree of if-then decisions. Interpretable and captures interactions but overfits without pruning.

An ensemble of decision trees trained on bootstrap samples with random feature subsets at each split, then averaged. Reduces variance relative to a single tree and is robust with little tuning.

An ensemble method that builds models sequentially, each new model fitting the residual errors of the current ensemble via gradient descent in function space. Strong on tabular data but sensitive to learning rate and depth.

A classifier that finds the separating hyperplane maximizing the margin between classes, with slack variables for non-separable data and kernel functions for nonlinear boundaries.

An unsupervised algorithm that partitions points into k clusters by alternating assignment to the nearest centroid and recomputation of centroids. Requires specifying k and assumes roughly spherical clusters.

A linear dimensionality reduction technique that projects data onto orthogonal directions of maximum variance, found via eigenvectors of the covariance matrix or singular value decomposition.

A cluster quality measure comparing each point's average distance to its own cluster versus the nearest other cluster, ranging from -1 to 1. Used to compare clusterings and choose k.

A loss measuring the dissimilarity between predicted probability distributions and true labels, equivalent to negative log-likelihood for categorical outcomes. Standard for classification networks.

The algorithm that computes gradients of a loss with respect to network parameters by applying the chain rule backward through the computation graph, enabling gradient-based training.

A training problem where gradients shrink toward zero as they propagate through many layers, stalling learning in early layers. Mitigated by residual connections, normalization, and appropriate activation functions.

A layer that normalizes activations across a mini-batch to stabilize and accelerate training, then applies learnable scale and shift. Behavior differs between training and inference modes.

A regularization technique that randomly zeroes a fraction of activations during training, preventing units from co-adapting. Disabled at inference, where activations are rescaled accordingly.

Degradation of model performance in production as the relationship between inputs and targets changes or input distributions shift away from training data. Detected by monitoring live metrics and feature distributions.

Running a new model alongside the production model on live traffic without serving its outputs, to compare behavior and performance safely before promotion.

A centralized system that computes, stores, and serves feature values consistently for both training and online inference, reducing training-serving skew and duplicated pipeline work.

A quantitative criterion for comparing model outcomes across groups, such as demographic parity, equalized odds, or calibration by group. Different criteria can be mutually incompatible.

A situation where a model or policy produces substantially different favorable outcome rates across protected groups, often assessed with a ratio of selection rates against a legal threshold.