DE MLO cheat sheet

An experiment run is identified by a run_id plus a hash of code version, data version, and hyperparameters; two runs with identical triples should produce identical metrics within hardware nondeterminism.

A model registry entry needs four immutable fields: model name, version, source run_id, and artifact URI; stage labels (Staging, Production, Archived) are mutable metadata and must never be the only deployment pointer.

Lineage rule: every registered model version must resolve to a training dataset snapshot and a commit SHA; if either is missing, the version is not reproducible and must be flagged.

Metric logging granularity: log per-step metrics for curves and per-epoch aggregates for comparison tables; per-step values are high-cardinality and should be downsampled before storage.

Artifact vs metadata split: store large artifacts (weights, checkpoints) in object storage and keep only URIs, hashes, and scalar metrics in the tracking store.

Promotion gate pattern: a candidate version moves to Production only after passing evaluation thresholds on a held-out set and a shadow or canary comparison against the current Production version.

Rollback requires retaining the previous N Production artifacts; a common rule is N>=2 model versions plus their preprocessing pipelines.

Checkpoint policy: save at fixed intervals and on best-metric improvement; keep the best checkpoint plus the last checkpoint to allow both resume and final export.

Bias-variance read: high training error with high validation error indicates underfitting (raise capacity or train longer); low training error with high validation error indicates overfitting (regularize, augment, or add data).

Learning-rate diagnostics: loss diverging or NaN within the first epochs usually means the learning rate is too high; near-flat loss means too low. Warmup plus cosine decay is a common stable schedule.

Effective batch size with gradient accumulation: effective_batch = micro_batch * accumulation_steps * data_parallel_workers.

Data leakage rule: fit scalers, encoders, and target statistics on the training split only, then apply the fitted transform to validation and test splits.

Cross-validation choice: k-fold for i.i.d. tabular data; grouped or time-series splits when rows share entities or temporal order, otherwise scores are optimistically biased.

Hyperparameter search: random search dominates grid search when only a few hyperparameters matter; Bayesian or successive-halving search is preferred when each trial is expensive.

Class imbalance handling order: adjust class weights or sampling first, then tune the decision threshold on validation data; never tune the threshold on the test set.

Early stopping uses a validation metric with a patience window and restores the best checkpoint, not the last epoch's weights.

Online vs batch decision: use batch scoring when freshness requirements exceed minutes and predictions can be precomputed; use online serving when the request needs features not known in advance.

Latency budget decomposition: total_p95 = queueing + preprocessing + inference + postprocessing; the model's share is usually the smallest part, so profile the whole path.

Model server batching trades throughput for latency: increasing max_batch_size raises throughput and p95 latency; set a max queue delay to bound the wait.

Deployment strategies: canary sends a small traffic fraction to the new version; blue-green swaps entire traffic and allows instant rollback by switching back.

Inference parity check: the serving preprocessing must be the same code path as training preprocessing; duplicated logic is the most common source of train-serve skew.

Resource sizing: measure peak memory including weights, activations, and KV cache for generative models; GPU memory must cover all three or requests will OOM under load.

Contract validation: reject requests failing schema checks at the gateway with a 4xx before they reach the model to protect the inference workers.

Autoscaling signals: scale on queue depth or concurrent requests rather than CPU alone, because GPU-bound servers show low CPU while saturated.

Data drift measures a change in input distribution P(X); concept drift measures a change in P(Y|X). Only concept drift necessarily degrades a supervised model's accuracy.

Population Stability Index: PSI = sum over bins of (actual% - expected%) * ln(actual% / expected%). Common thresholds: PSI < 0.1 stable, 0.1-0.25 moderate shift, > 0.25 significant shift.

Kullback-Leibler divergence is asymmetric and undefined when a bin has zero expected probability; use smoothing or Jensen-Shannon divergence for a symmetric bounded alternative.

When labels arrive late, monitor proxy signals: prediction distribution, feature drift, and confidence histograms; schedule delayed label joins to compute true performance.

Alerting needs a baseline window and a minimum sample size per window; a drift alert on a few hundred rows is usually noise.

Performance decay triage order: verify data pipeline integrity, then feature drift, then concept drift, then upstream model or dependency changes.

Retraining triggers should be explicit: scheduled cadence, drift threshold breach, or performance below an SLA, each with an owner and a validation gate.

Reproducibility requires pinning four things: code commit, dataset version or snapshot hash, environment (dependency lockfile plus container digest), and random seeds.

Seed control: set seeds for Python, NumPy, and the framework; GPU nondeterminism may still require deterministic algorithm flags at a performance cost.

Model card contents: intended use, out-of-scope uses, training data description, evaluation metrics by subgroup, and known limitations.

Dataset documentation should record provenance, collection method, consent or licensing, labeling process, and known biases.

Audit trail requirement: every Production prediction should be traceable to a model version and the feature values used, retained per the applicable retention policy.

Access control: separate roles for experimenters, approvers, and deployers; a single person holding both training and Production promotion rights weakens segregation of duties.

Approval gate evidence: evaluation report, bias and fairness checks for affected subgroups, and a rollback plan must exist before a model version enters Production.