MS MLO key terms

A centralized system that stores curated feature definitions and serves the same computed values to both offline training pipelines and online inference endpoints, preventing training-serving skew.

A mismatch between how features are computed during model training and how they are computed at inference time, which silently degrades production accuracy even when offline metrics look strong.

The property that a training feature value reflects only data available at or before the label's timestamp, preventing future information from leaking into historical training rows.

A modeling error where information unavailable at prediction time, such as a post-outcome column or future record, enters training features and inflates validation scores unrealistically.

A specific form of leakage where a feature is derived from or strongly encodes the label itself, producing near-perfect offline results that collapse when the model is deployed.

Tracking immutable snapshots or content hashes of datasets and schemas so any model can be retrained from the exact data it was originally trained on.

A recorded graph of how data, features, code, and artifacts flow into a model and its predictions, used for debugging, auditing, and impact analysis.

Logging of parameters, metrics, artifacts, and code versions for each training run so results can be compared, reproduced, and promoted into a registry.

A versioned catalog of trained model artifacts with metadata, stage or alias assignments, and approval state, acting as the handoff point between training and deployment pipelines.

The search over configuration values not learned from data, such as learning rate or tree depth, using strategies like grid, random, Bayesian, or early-stopping searches.

A resampling technique that partitions data into folds, training on some and validating on others repeatedly, to estimate generalization performance more reliably than a single split.

Data withheld entirely from training and tuning and used once for a final unbiased performance estimate before a model is considered for release.

When a model fits noise and idiosyncrasies of training data so closely that it performs well in-sample but poorly on new, unseen examples.

Techniques such as weight penalties, dropout, or early stopping that constrain model complexity to improve generalization to unseen data.

Scoring large volumes of records on a schedule or trigger, writing predictions to storage for downstream consumption rather than answering individual requests in real time.

Serving predictions synchronously per request through an endpoint, with strict latency budgets and availability requirements typical of interactive applications.

Scoring records continuously as they arrive on an event stream, emitting predictions with low delay while decoupling producers and consumers.

A deployed, addressable service that loads a model artifact and returns predictions for incoming requests, often behind a gateway that handles authentication and routing.

Automatically adjusting the number of serving replicas based on traffic or latency signals so capacity tracks demand without manual intervention.

Releasing a new model version to a small slice of live traffic first, then gradually expanding exposure while comparing health and quality metrics against the incumbent.

Running two identical environments where traffic switches from the current version to the new one at once, enabling fast rollback by switching back.

Sending production requests to a candidate model in parallel with the live model while discarding its outputs, to observe behavior without affecting users.

A pattern where the current production model (champion) is compared against one or more candidate models (challengers) on live or replayed traffic to decide promotion.

Reverting a deployment to a previously known-good model version and configuration when the new release shows errors, latency regressions, or quality degradation.

A change over time in the distribution of input features relative to the training distribution, which can degrade predictions even when the underlying relationship is unchanged.

A change in the true relationship between inputs and the target, so the same feature values now imply different outcomes and the model's learned mapping becomes stale.

Rebuilding or updating a model with fresher data or code, triggered on a schedule or by drift and performance alerts, then revalidated before redeployment.

Continuous collection of operational and quality signals such as latency, error rates, prediction distributions, and drift scores to detect when a deployed model needs attention.

The delay between a prediction and the arrival of its verified label, which means true quality metrics in production are always measured on older data.

An indirect signal, such as prediction confidence or input drift, used to approximate model quality when verified labels are unavailable or arrive too late.

A quantitative measure of outcome parity across groups, such as demographic parity or equalized odds, used to assess whether a model disadvantages protected populations.

Interventions applied before, during, or after training, such as rebalancing data, adding fairness constraints, or adjusting thresholds, to reduce disparate model outcomes.

A short structured document describing a model's intended use, training data, evaluation results, limitations, and ethical considerations for downstream consumers and reviewers.

Detecting and removing or masking personally identifiable information in logs, datasets, and prompts so sensitive identifiers are not exposed to unauthorized systems or people.

Granting identities only the minimum permissions needed for their tasks, such as read-only dataset access for training jobs and separate credentials for deployment pipelines.

Storing API keys, tokens, and credentials in a dedicated vault with rotation and audit, never hard-coded in notebooks, repositories, or container images.

A non-human identity used by automated pipelines and applications to authenticate to cloud resources, scoped with roles rather than shared user credentials.

Automated stages that build, test, package, and release code and model artifacts, giving repeatable, auditable promotion from commit to production.

Declaring compute, storage, networking, and serving resources in version-controlled configuration files so environments are reproducible and changes are reviewable.

A low-latency lookup layer that returns precomputed feature values for a single entity at request time, typically backed by a key-value store.