ICML flashcards

35 cards. Read the term and the definition here. The deck at the top of this page is optional.

Bias-variance decomposition

Expected squared prediction error equals squared bias plus variance plus irreducible noise; increasing model complexity typically lowers bias but raises variance.

Bayes' theorem

P(A|B) = P(B|A)P(A)/P(B). The posterior probability is proportional to the likelihood of the evidence times the prior probability.

Central limit theorem

For independent samples with finite variance, the sampling distribution of the sample mean approaches a normal distribution with mean mu and variance sigma squared over n.

L2 regularization

Adds lambda times the squared L2 norm of the weight vector to the loss, shrinking coefficients toward zero and reducing variance without producing exact zeros.

p-value definition

The probability of observing a test statistic at least as extreme as the sample value, assuming the null hypothesis is true; not the probability the null is true.

Target leakage

A feature contains information about the label that would not be available at prediction time, producing unrealistically strong validation performance that fails in production.

One-hot encoding

Represents a categorical variable with k levels as k binary indicator columns; dropping one column avoids perfect collinearity in linear models.

Standardization versus min-max

Standardization maps values to z-scores using mean and standard deviation; min-max scaling maps values linearly into a fixed range such as zero to one.

Variance inflation factor

VIF equals one over one minus R-squared from regressing a feature on the others; values above about ten indicate problematic multicollinearity.

Class imbalance handling

Options include resampling the minority or majority class, assigning class weights in the loss, and tuning the decision threshold instead of relying on accuracy.

Logistic regression

Models the log-odds of a binary outcome as a linear function of features and is trained by minimizing binary cross-entropy loss.

Random forest variance reduction

Averages many decision trees trained on bootstrap samples with random feature subsets, decorrelating trees and lowering ensemble variance compared to a single tree.

Gradient boosting

Builds trees sequentially, each fit to the negative gradient of the loss with respect to current predictions, gradually reducing training error.

SVM margin and C

Support vector machines maximize the margin between classes; the C parameter trades off margin width against tolerance of training misclassification.

k-means objective

Minimizes the within-cluster sum of squared distances to cluster centroids; requires choosing k and is sensitive to initialization, motivating k-means++.

Principal component analysis

Projects data onto orthogonal directions ordered by explained variance, which are the top eigenvectors of the covariance matrix; used for compression and visualization.

DBSCAN parameters

Groups points in dense neighborhoods defined by radius eps and minimum points minPts; points in sparse regions are labeled noise and clusters can be non-convex.

Silhouette score

Compares a point's mean distance to its own cluster versus the nearest other cluster; ranges from minus one to one, with higher values indicating better separation.

Precision and recall

Precision is true positives over predicted positives; recall is true positives over actual positives; F1 is their harmonic mean.

ROC-AUC interpretation

The probability that a randomly chosen positive example receives a higher score than a randomly chosen negative example; threshold-independent and insensitive to class balance.

Stratified k-fold cross-validation

Splits data into k folds while preserving class proportions in each fold, giving more stable performance estimates on imbalanced datasets.

Data leakage in validation

Preprocessing steps such as scaling or feature selection fitted on the full dataset before splitting leak test information and inflate cross-validation estimates.

Probability calibration

Ensures predicted probabilities match observed frequencies; assessed with reliability curves and corrected by Platt scaling or isotonic regression.

Backpropagation

Computes loss gradients for each parameter by applying the chain rule backward through the network, enabling gradient-based weight updates.

Batch normalization

Normalizes layer activations using mini-batch mean and variance during training, stabilizing optimization and allowing higher learning rates; uses running statistics at inference.

Dropout regularization

Randomly sets a fraction of activations to zero during training, preventing co-adaptation of units; it is disabled at inference time.

Adam optimizer

Combines momentum with adaptive learning rates by tracking exponential moving averages of gradients and squared gradients, with bias correction for early steps.

Self-attention mechanism

Computes softmax of query-key dot products scaled by the square root of key dimension, then weights values, letting each token attend to all others.

Training-serving skew

A mismatch between feature computation in the training pipeline and the live inference path, causing production performance to degrade relative to offline metrics.

Canary deployment

Routes a small fraction of live traffic to a new model version while monitoring error and latency, limiting blast radius before full rollout.

Concept drift

The statistical relationship between input features and the target changes over time, so a deployed model's performance degrades and retraining is needed.

Demographic parity

A fairness criterion requiring equal positive prediction rates across protected groups, regardless of underlying base rates or true label distributions.

Equalized odds

Requires equal true positive rates and equal false positive rates across protected groups, conditioning fairness on the actual outcome.

SHAP values

Shapley-value-based feature attributions from cooperative game theory that explain an individual prediction as additive contributions summing to the model output.

Model card

A structured document describing a model's intended use, training data, evaluation results, limitations, and ethical considerations for transparency and governance.