DE MLO flashcards

33 cards. Read the term and the definition here. The deck at the top of this page is optional.

Run reproducibility triple

Code commit, data version, and hyperparameter set. If all three match, a rerun should reproduce metrics within expected hardware nondeterminism, making the run auditable.

Model registry version

An immutable record holding model name, version number, originating run_id, and artifact URI. Stage labels are mutable and must not be the sole deployment pointer.

Lineage completeness check

A registered model version is reproducible only if it resolves to both a training dataset snapshot and a source commit. Missing either makes the version untraceable.

Artifact versus metadata storage

Large files such as weights and checkpoints live in object storage; the tracking store keeps URIs, hashes, and scalar metrics. This keeps the tracking backend fast.

Promotion gate

A rule requiring a candidate version to pass evaluation thresholds on held-out data and a shadow or canary comparison before it replaces the current Production version.

Checkpoint retention policy

Keep the best-metric checkpoint and the most recent checkpoint at minimum, so training can resume after failure and the final model can still be exported.

Overfitting signature

Low training error combined with high validation error. Remedies include regularization, data augmentation, more data, or reduced model capacity.

Underfitting signature

High training error and high validation error together. The model lacks capacity or training time; increase model size, features, or epochs.

Effective batch size formula

effective_batch = micro_batch * accumulation_steps * data_parallel_workers. It determines the learning-rate scaling needed when memory limits force small micro-batches.

Data leakage prevention

Fit scalers, encoders, and target statistics only on the training split, then apply the fitted transforms to validation and test data.

Grouped cross-validation

A split scheme that keeps all rows of an entity in the same fold. Required when rows share users, patients, or devices, otherwise scores are optimistically biased.

Random versus grid search

Random search covers more of the important hyperparameter space when only a few parameters matter. Grid search wastes trials on irrelevant dimensions.

Early stopping restore

Stopping on a validation metric with a patience window and restoring the best checkpoint, not the final epoch weights, which may already be overfit.

Threshold tuning placement

Tune the decision threshold for imbalanced classes on validation data only. Tuning on the test set leaks information and inflates reported performance.

Online versus batch serving

Batch scoring precomputes predictions when minute-level freshness is enough. Online serving is needed when inputs are only known at request time.

Latency budget decomposition

total_p95 = queueing + preprocessing + inference + postprocessing. Profiling the whole path matters because inference is often the smallest component.

Server-side batching tradeoff

Larger max_batch_size raises throughput but increases p95 latency. A max queue delay bounds how long a request waits to join a batch.

Canary deployment

A small traffic fraction is routed to the new model version while metrics are compared against the current version; the fraction increases on success.

Blue-green deployment

Two identical environments where traffic switches entirely from old to new. Rollback is instant because the previous environment remains intact.

Train-serve skew

A mismatch between training and serving preprocessing. Prevented by sharing one preprocessing code path or artifact rather than duplicating logic.

GPU memory sizing

Peak GPU memory must cover model weights, activations, and KV cache for generative models. Underestimating any component causes out-of-memory errors under load.

Request schema validation

Gateway-level checks that reject malformed requests with a 4xx before they reach inference workers, protecting capacity and keeping error rates clean.

Data drift versus concept drift

Data drift changes the input distribution P(X). Concept drift changes the relationship P(Y|X). Only concept drift necessarily degrades supervised model accuracy.

Population Stability Index formula

PSI = sum over bins of (actual% - expected%) * ln(actual% / expected%). Below 0.1 is stable; above 0.25 indicates significant distribution shift.

Jensen-Shannon divergence

A symmetric, bounded divergence used to compare distributions when KL divergence is unsuitable because it is asymmetric or hits zero-probability bins.

Proxy monitoring without labels

When ground truth arrives late, monitor prediction distributions, feature drift, and confidence histograms, then join delayed labels to compute true performance.

Drift alert minimum sample

Drift statistics need a baseline window and a minimum row count per window. Alerts computed on very small samples are usually noise.

Performance decay triage order

Check pipeline integrity first, then feature drift, then concept drift, then upstream dependency changes. This order avoids retraining for a data bug.

Retraining trigger

An explicit condition such as a scheduled cadence, a drift threshold breach, or performance below SLA, each with a named owner and validation gate.

Reproducibility four pins

Code commit, dataset version, environment lockfile plus container digest, and random seeds. All four must be pinned for a run to be reproducible.

Deterministic GPU flags

Framework flags that force deterministic algorithms. They make reruns bitwise stable but reduce throughput, so they are usually enabled only for audits.

Model card contents

Intended use, out-of-scope uses, training data description, evaluation metrics by subgroup, and known limitations. It documents a model for downstream consumers.

Segregation of duties

Separate roles for experimenting, approving, and deploying. One person holding both training and promotion rights weakens governance controls.