MS MLO cheat sheet

Problem framing decision rule: classification when the target is discrete and labeled, regression when continuous, clustering when no labels exist; time-series forecasting requires temporal validation splits, never random splits.

Azure ML workspace is the top-level resource; compute (compute instance for dev, compute cluster for training/batch, inference cluster or managed online endpoint for serving), datastores, and registered assets are children of the workspace.

Managed online endpoints suit low-latency real-time scoring with autoscaling; batch endpoints suit large-file, long-running jobs; Kubernetes online endpoints are chosen when you need full cluster control.

Latency budget rule: real-time endpoints target sub-second per-request scoring; if preprocessing exceeds the SLA, move featurization into the model pipeline or a precompute step.

Design for reproducibility by pinning environment images (conda/pip lock), random seeds, and dataset versions; an MLflow run records parameters, metrics, and artifacts for each experiment.

Cost rule: use low-priority (spot) compute clusters for interruptible training, autoscale-to-zero for dev compute, and separate training from inference compute so idle inference nodes can scale down.

Idempotency requirement: training jobs and deployment scripts must be safe to re-run; use unique run names derived from code hash plus data version.

Capacity planning rule: size inference compute from peak requests per second times average latency, then add headroom; autoscale instance count with a minimum of one for production endpoints.

Data asset types: File dataset (any file path, no schema) vs Table (tabular, schema-enforced, supports MLTable for typed, lazy loading with schema validation).

Data drift vs concept drift: data drift is a change in input distribution P(X); concept drift is a change in the relationship P(Y|X); drift detection on inputs will not catch concept drift alone.

Leakage rule: any feature computed using information unavailable at prediction time (target encoding over the full set, future values, post-event fields) must be removed or computed inside the training fold.

Feature store purpose: single definition of a feature reused across training and inference, with offline store for batch training and online store for low-latency lookup at scoring time.

Time-based features require point-in-time correctness; joins must use the feature value as of the event timestamp, not the latest value, to avoid look-ahead bias.

Missing data decision rule: drop a column when missingness is high and uninformative; impute with median/mode plus a missing-indicator when missingness is informative; never impute using data from the validation or test split.

Cardinality rule: high-cardinality categoricals need hashing, frequency capping, or target encoding computed within folds; one-hot encoding explodes dimensionality and is avoided above a few hundred levels.

Data versioning: register datasets with a version and reference the immutable version in training jobs; never train against a mutable path that changes under the same name.

Hyperparameter search strategies: grid search cost grows multiplicatively with dimensions; random search covers high-dimensional spaces better per trial; Bayesian (TPE) search uses prior trials and is preferred when each trial is expensive.

Hyperdrive in Azure ML defines search space, sampling algorithm (random, grid, Bayesian), primary metric with goal, early-termination policy (bandit, median stopping), and max total/parallel trials.

Early-termination policies stop underperforming runs early: bandit uses a slack factor against the best run; median stopping compares against the running median of the primary metric.

Overfitting signals: training loss keeps falling while validation loss rises, large gap between train and validation metrics, and high variance across cross-validation folds.

Cross-validation rule: use k-fold for i.i.d. data, stratified k-fold for class imbalance, group k-fold when records cluster (patient, user), and time-series split for temporal data.

Regularization levers: L1 (sparsity, feature selection), L2 (shrinkage, weight decay), dropout, early stopping, and reduced model capacity; tune strength on validation, not test.

Imbalanced classification: prefer PR-AUC over ROC-AUC when positives are rare; apply class weights or resampling inside the training fold only, never before the split.

Experiment tracking: log parameters, metrics, and artifacts per run; compare runs in the experiment view; register the best run's model as a versioned asset with tags for the training data version.

Deployment modes: managed online endpoint (real-time, autoscale, key or token auth), batch endpoint (invoke over a data asset, writes scored output), and Kubernetes online endpoint (self-managed cluster).

Blue-green vs canary: blue-green swaps all traffic at once for instant rollback; canary splits a small percentage of traffic to the new deployment and increases gradually.

A deployment is a versioned instance of a model plus environment plus scoring script; an endpoint routes traffic to one or more deployments by percentage, enabling safe rollout.

Scoring script contract: init() loads the model once at startup; run(raw_data) receives JSON and must return JSON-serializable predictions; heavy work belongs in init, not run.

Environment rule: pin base image, Python version, and package versions in the environment; the model and environment are immutable inputs to a deployment, so a change requires a new deployment.

Model formats: MLflow no-code deployment needs an MLmodel file with signature; custom deployments need a scoring script; ONNX is chosen for cross-framework runtime portability.

Auth and networking: endpoints support key-based and Azure AD token auth; private endpoints and virtual network integration are used when traffic must not traverse the public internet.

Rollback rule: keep the previous deployment alive at zero or low traffic so rollback is a traffic-percentage change, not a rebuild.

Model monitoring signals: data drift on inputs, prediction drift on outputs, data quality (nulls, type, range violations), and model performance when ground truth arrives with delay.

Drift metrics: population stability index (PSI) and KL divergence for distribution shift; PSI above roughly 0.2 signals meaningful shift, above 0.25 often triggers retraining.

Performance decay detection requires labeled outcomes; when labels lag, monitor proxy signals (input drift, prediction distribution, feature null rate) and schedule delayed evaluation.

Application Insights integration captures endpoint request logs, latency, error rates, and dependency calls; alerts on latency percentiles (p95, p99) rather than averages.

Retraining triggers: scheduled cadence, drift threshold breach, performance below SLA, or data pipeline schema change; retraining must reuse the same evaluation gates as the original model.

Observability rule: log the model version, deployment name, and request ID with each prediction so incidents can be traced to a specific artifact.

Alert design: alert on symptoms (error rate, latency, drift score) with severity tiers; avoid alerting on every metric to prevent fatigue; include runbook links in alert payloads.

Data collection for monitoring: enable endpoint data collection to a storage target, respecting privacy rules; sample high-volume traffic rather than logging every payload.

Fairness metrics: demographic parity compares positive rates across groups; equalized odds compares true/false positive rates; choose the metric that matches the harm and the legal context, and report disaggregated metrics.

Interpretability tools: global feature importance (permutation, SHAP summary) explains the model overall; local explanations (SHAP force/waterfall, LIME) explain a single prediction; use for debugging and for adverse-action explanations.

Error analysis: slice evaluation by sensitive and business-relevant subgroups; an aggregate metric can hide a subgroup with materially worse performance.

Privacy: differential privacy adds calibrated noise to protect individuals; k-anonymity and generalization reduce re-identification; never train on PII without a lawful basis and minimization.

Security controls: workspace uses Azure RBAC and managed identities; secrets live in Key Vault, never in code or environment files; compute sits in a virtual network with private endpoints.

Model governance artifacts: model card (intended use, limitations, metrics, ethical considerations), datasheet for datasets, and an audit trail of who approved which model version.

Human oversight rule: high-impact decisions require a human-in-the-loop review path and a documented appeal or override mechanism.

Responsible AI checklist gates: fairness testing, interpretability review, privacy impact assessment, and harm/limitation documentation before production approval.

Two pipelines: CI validates code, runs unit tests, and builds the environment image; CD registers the model and deploys to an endpoint, gated by evaluation metrics.

Azure ML CLI v2 and Python SDK v2 express jobs, components, and pipelines as YAML; components are reusable, versioned steps with typed inputs and outputs.

Model registration gate rule: promote a model only if it beats the current production model on the primary metric and passes fairness and drift checks on a holdout set.

Infrastructure as code: define workspace, compute, datastores, and endpoints in Bicep or Terraform; keep environment definitions in version control so deployments are reproducible.

Trigger design: CI on pull request, CD on merge to main or on model registration event; use separate dev/test/prod workspaces and promote artifacts, not retrain per environment.

Automation of retraining: schedule a pipeline with a time trigger or a data-change trigger; the pipeline must include data validation, training, evaluation, and conditional registration.

GitHub Actions or Azure Pipelines authenticate to Azure ML with a service principal or federated credential (OIDC) stored as a secret reference, not a plaintext secret.

Rollback automation: CD keeps the previous deployment addressable so a failed smoke test triggers an automatic traffic shift back.

Problem framing decision rule: classification when the target is discrete and labeled, regression when continuous, clustering when no labels exist; time-series forecasting requires temporal validation splits, never random splits.

Azure ML workspace is the top-level resource; compute (compute instance for dev, compute cluster for training/batch, inference cluster or managed online endpoint for serving), datastores, and registered assets are children of the workspace.

Managed online endpoints suit low-latency real-time scoring with autoscaling; batch endpoints suit large-file, long-running jobs; Kubernetes online endpoints are chosen when you need full cluster control.

Latency budget rule: real-time endpoints target sub-second per-request scoring; if preprocessing exceeds the SLA, move featurization into the model pipeline or a precompute step.

Design for reproducibility by pinning environment images (conda/pip lock), random seeds, and dataset versions; an MLflow run records parameters, metrics, and artifacts for each experiment.

Cost rule: use low-priority (spot) compute clusters for interruptible training, autoscale-to-zero for dev compute, and separate training from inference compute so idle inference nodes can scale down.

Idempotency requirement: training jobs and deployment scripts must be safe to re-run; use unique run names derived from code hash plus data version.

Capacity planning rule: size inference compute from peak requests per second times average latency, then add headroom; autoscale instance count with a minimum of one for production endpoints.

Data asset types: File dataset (any file path, no schema) vs Table (tabular, schema-enforced, supports MLTable for typed, lazy loading with schema validation).

Data drift vs concept drift: data drift is a change in input distribution P(X); concept drift is a change in the relationship P(Y|X); drift detection on inputs will not catch concept drift alone.

Leakage rule: any feature computed using information unavailable at prediction time (target encoding over the full set, future values, post-event fields) must be removed or computed inside the training fold.

Feature store purpose: single definition of a feature reused across training and inference, with offline store for batch training and online store for low-latency lookup at scoring time.

Time-based features require point-in-time correctness; joins must use the feature value as of the event timestamp, not the latest value, to avoid look-ahead bias.

Missing data decision rule: drop a column when missingness is high and uninformative; impute with median/mode plus a missing-indicator when missingness is informative; never impute using data from the validation or test split.

Cardinality rule: high-cardinality categoricals need hashing, frequency capping, or target encoding computed within folds; one-hot encoding explodes dimensionality and is avoided above a few hundred levels.

Data versioning: register datasets with a version and reference the immutable version in training jobs; never train against a mutable path that changes under the same name.

Hyperparameter search strategies: grid search cost grows multiplicatively with dimensions; random search covers high-dimensional spaces better per trial; Bayesian (TPE) search uses prior trials and is preferred when each trial is expensive.

Hyperdrive in Azure ML defines search space, sampling algorithm (random, grid, Bayesian), primary metric with goal, early-termination policy (bandit, median stopping), and max total/parallel trials.

Early-termination policies stop underperforming runs early: bandit uses a slack factor against the best run; median stopping compares against the running median of the primary metric.

Overfitting signals: training loss keeps falling while validation loss rises, large gap between train and validation metrics, and high variance across cross-validation folds.

Cross-validation rule: use k-fold for i.i.d. data, stratified k-fold for class imbalance, group k-fold when records cluster (patient, user), and time-series split for temporal data.

Regularization levers: L1 (sparsity, feature selection), L2 (shrinkage, weight decay), dropout, early stopping, and reduced model capacity; tune strength on validation, not test.

Imbalanced classification: prefer PR-AUC over ROC-AUC when positives are rare; apply class weights or resampling inside the training fold only, never before the split.

Experiment tracking: log parameters, metrics, and artifacts per run; compare runs in the experiment view; register the best run's model as a versioned asset with tags for the training data version.

Deployment modes: managed online endpoint (real-time, autoscale, key or token auth), batch endpoint (invoke over a data asset, writes scored output), and Kubernetes online endpoint (self-managed cluster).

Blue-green vs canary: blue-green swaps all traffic at once for instant rollback; canary splits a small percentage of traffic to the new deployment and increases gradually.

A deployment is a versioned instance of a model plus environment plus scoring script; an endpoint routes traffic to one or more deployments by percentage, enabling safe rollout.

Scoring script contract: init() loads the model once at startup; run(raw_data) receives JSON and must return JSON-serializable predictions; heavy work belongs in init, not run.

Environment rule: pin base image, Python version, and package versions in the environment; the model and environment are immutable inputs to a deployment, so a change requires a new deployment.

Model formats: MLflow no-code deployment needs an MLmodel file with signature; custom deployments need a scoring script; ONNX is chosen for cross-framework runtime portability.

Auth and networking: endpoints support key-based and Azure AD token auth; private endpoints and virtual network integration are used when traffic must not traverse the public internet.

Rollback rule: keep the previous deployment alive at zero or low traffic so rollback is a traffic-percentage change, not a rebuild.

Model monitoring signals: data drift on inputs, prediction drift on outputs, data quality (nulls, type, range violations), and model performance when ground truth arrives with delay.

Drift metrics: population stability index (PSI) and KL divergence for distribution shift; PSI above roughly 0.2 signals meaningful shift, above 0.25 often triggers retraining.

Performance decay detection requires labeled outcomes; when labels lag, monitor proxy signals (input drift, prediction distribution, feature null rate) and schedule delayed evaluation.

Application Insights integration captures endpoint request logs, latency, error rates, and dependency calls; alerts on latency percentiles (p95, p99) rather than averages.

Retraining triggers: scheduled cadence, drift threshold breach, performance below SLA, or data pipeline schema change; retraining must reuse the same evaluation gates as the original model.

Observability rule: log the model version, deployment name, and request ID with each prediction so incidents can be traced to a specific artifact.

Alert design: alert on symptoms (error rate, latency, drift score) with severity tiers; avoid alerting on every metric to prevent fatigue; include runbook links in alert payloads.

Data collection for monitoring: enable endpoint data collection to a storage target, respecting privacy rules; sample high-volume traffic rather than logging every payload.

Fairness metrics: demographic parity compares positive rates across groups; equalized odds compares true/false positive rates; choose the metric that matches the harm and the legal context, and report disaggregated metrics.

Interpretability tools: global feature importance (permutation, SHAP summary) explains the model overall; local explanations (SHAP force/waterfall, LIME) explain a single prediction; use for debugging and for adverse-action explanations.

Error analysis: slice evaluation by sensitive and business-relevant subgroups; an aggregate metric can hide a subgroup with materially worse performance.

Privacy: differential privacy adds calibrated noise to protect individuals; k-anonymity and generalization reduce re-identification; never train on PII without a lawful basis and minimization.

Security controls: workspace uses Azure RBAC and managed identities; secrets live in Key Vault, never in code or environment files; compute sits in a virtual network with private endpoints.

Model governance artifacts: model card (intended use, limitations, metrics, ethical considerations), datasheet for datasets, and an audit trail of who approved which model version.

Human oversight rule: high-impact decisions require a human-in-the-loop review path and a documented appeal or override mechanism.

Responsible AI checklist gates: fairness testing, interpretability review, privacy impact assessment, and harm/limitation documentation before production approval.

Two pipelines: CI validates code, runs unit tests, and builds the environment image; CD registers the model and deploys to an endpoint, gated by evaluation metrics.

Azure ML CLI v2 and Python SDK v2 express jobs, components, and pipelines as YAML; components are reusable, versioned steps with typed inputs and outputs.

Model registration gate rule: promote a model only if it beats the current production model on the primary metric and passes fairness and drift checks on a holdout set.

Infrastructure as code: define workspace, compute, datastores, and endpoints in Bicep or Terraform; keep environment definitions in version control so deployments are reproducible.

Trigger design: CI on pull request, CD on merge to main or on model registration event; use separate dev/test/prod workspaces and promote artifacts, not retrain per environment.

Automation of retraining: schedule a pipeline with a time trigger or a data-change trigger; the pipeline must include data validation, training, evaluation, and conditional registration.

GitHub Actions or Azure Pipelines authenticate to Azure ML with a service principal or federated credential (OIDC) stored as a secret reference, not a plaintext secret.

Rollback automation: CD keeps the previous deployment addressable so a failed smoke test triggers an automatic traffic shift back.

Problem framing decision rule: classification when the target is discrete and labeled, regression when continuous, clustering when no labels exist; time-series forecasting requires temporal validation splits, never random splits.

Azure ML workspace is the top-level resource; compute (compute instance for dev, compute cluster for training/batch, inference cluster or managed online endpoint for serving), datastores, and registered assets are children of the workspace.

Managed online endpoints suit low-latency real-time scoring with autoscaling; batch endpoints suit large-file, long-running jobs; Kubernetes online endpoints are chosen when you need full cluster control.

Latency budget rule: real-time endpoints target sub-second per-request scoring; if preprocessing exceeds the SLA, move featurization into the model pipeline or a precompute step.

Design for reproducibility by pinning environment images (conda/pip lock), random seeds, and dataset versions; an MLflow run records parameters, metrics, and artifacts for each experiment.

Cost rule: use low-priority (spot) compute clusters for interruptible training, autoscale-to-zero for dev compute, and separate training from inference compute so idle inference nodes can scale down.

Idempotency requirement: training jobs and deployment scripts must be safe to re-run; use unique run names derived from code hash plus data version.

Capacity planning rule: size inference compute from peak requests per second times average latency, then add headroom; autoscale instance count with a minimum of one for production endpoints.

Data asset types: File dataset (any file path, no schema) vs Table (tabular, schema-enforced, supports MLTable for typed, lazy loading with schema validation).

Data drift vs concept drift: data drift is a change in input distribution P(X); concept drift is a change in the relationship P(Y|X); drift detection on inputs will not catch concept drift alone.

Leakage rule: any feature computed using information unavailable at prediction time (target encoding over the full set, future values, post-event fields) must be removed or computed inside the training fold.

Feature store purpose: single definition of a feature reused across training and inference, with offline store for batch training and online store for low-latency lookup at scoring time.

Time-based features require point-in-time correctness; joins must use the feature value as of the event timestamp, not the latest value, to avoid look-ahead bias.

Missing data decision rule: drop a column when missingness is high and uninformative; impute with median/mode plus a missing-indicator when missingness is informative; never impute using data from the validation or test split.

Cardinality rule: high-cardinality categoricals need hashing, frequency capping, or target encoding computed within folds; one-hot encoding explodes dimensionality and is avoided above a few hundred levels.

Data versioning: register datasets with a version and reference the immutable version in training jobs; never train against a mutable path that changes under the same name.

Hyperparameter search strategies: grid search cost grows multiplicatively with dimensions; random search covers high-dimensional spaces better per trial; Bayesian (TPE) search uses prior trials and is preferred when each trial is expensive.

Hyperdrive in Azure ML defines search space, sampling algorithm (random, grid, Bayesian), primary metric with goal, early-termination policy (bandit, median stopping), and max total/parallel trials.

Early-termination policies stop underperforming runs early: bandit uses a slack factor against the best run; median stopping compares against the running median of the primary metric.

Overfitting signals: training loss keeps falling while validation loss rises, large gap between train and validation metrics, and high variance across cross-validation folds.

Cross-validation rule: use k-fold for i.i.d. data, stratified k-fold for class imbalance, group k-fold when records cluster (patient, user), and time-series split for temporal data.

Regularization levers: L1 (sparsity, feature selection), L2 (shrinkage, weight decay), dropout, early stopping, and reduced model capacity; tune strength on validation, not test.

Imbalanced classification: prefer PR-AUC over ROC-AUC when positives are rare; apply class weights or resampling inside the training fold only, never before the split.

Experiment tracking: log parameters, metrics, and artifacts per run; compare runs in the experiment view; register the best run's model as a versioned asset with tags for the training data version.

Deployment modes: managed online endpoint (real-time, autoscale, key or token auth), batch endpoint (invoke over a data asset, writes scored output), and Kubernetes online endpoint (self-managed cluster).

Blue-green vs canary: blue-green swaps all traffic at once for instant rollback; canary splits a small percentage of traffic to the new deployment and increases gradually.

A deployment is a versioned instance of a model plus environment plus scoring script; an endpoint routes traffic to one or more deployments by percentage, enabling safe rollout.

Scoring script contract: init() loads the model once at startup; run(raw_data) receives JSON and must return JSON-serializable predictions; heavy work belongs in init, not run.

Environment rule: pin base image, Python version, and package versions in the environment; the model and environment are immutable inputs to a deployment, so a change requires a new deployment.

Model formats: MLflow no-code deployment needs an MLmodel file with signature; custom deployments need a scoring script; ONNX is chosen for cross-framework runtime portability.

Auth and networking: endpoints support key-based and Azure AD token auth; private endpoints and virtual network integration are used when traffic must not traverse the public internet.

Rollback rule: keep the previous deployment alive at zero or low traffic so rollback is a traffic-percentage change, not a rebuild.

Model monitoring signals: data drift on inputs, prediction drift on outputs, data quality (nulls, type, range violations), and model performance when ground truth arrives with delay.

Drift metrics: population stability index (PSI) and KL divergence for distribution shift; PSI above roughly 0.2 signals meaningful shift, above 0.25 often triggers retraining.

Performance decay detection requires labeled outcomes; when labels lag, monitor proxy signals (input drift, prediction distribution, feature null rate) and schedule delayed evaluation.

Application Insights integration captures endpoint request logs, latency, error rates, and dependency calls; alerts on latency percentiles (p95, p99) rather than averages.

Retraining triggers: scheduled cadence, drift threshold breach, performance below SLA, or data pipeline schema change; retraining must reuse the same evaluation gates as the original model.

Observability rule: log the model version, deployment name, and request ID with each prediction so incidents can be traced to a specific artifact.

Alert design: alert on symptoms (error rate, latency, drift score) with severity tiers; avoid alerting on every metric to prevent fatigue; include runbook links in alert payloads.

Data collection for monitoring: enable endpoint data collection to a storage target, respecting privacy rules; sample high-volume traffic rather than logging every payload.

Fairness metrics: demographic parity compares positive rates across groups; equalized odds compares true/false positive rates; choose the metric that matches the harm and the legal context, and report disaggregated metrics.

Interpretability tools: global feature importance (permutation, SHAP summary) explains the model overall; local explanations (SHAP force/waterfall, LIME) explain a single prediction; use for debugging and for adverse-action explanations.

Error analysis: slice evaluation by sensitive and business-relevant subgroups; an aggregate metric can hide a subgroup with materially worse performance.

Privacy: differential privacy adds calibrated noise to protect individuals; k-anonymity and generalization reduce re-identification; never train on PII without a lawful basis and minimization.

Security controls: workspace uses Azure RBAC and managed identities; secrets live in Key Vault, never in code or environment files; compute sits in a virtual network with private endpoints.

Model governance artifacts: model card (intended use, limitations, metrics, ethical considerations), datasheet for datasets, and an audit trail of who approved which model version.

Human oversight rule: high-impact decisions require a human-in-the-loop review path and a documented appeal or override mechanism.

Responsible AI checklist gates: fairness testing, interpretability review, privacy impact assessment, and harm/limitation documentation before production approval.

Two pipelines: CI validates code, runs unit tests, and builds the environment image; CD registers the model and deploys to an endpoint, gated by evaluation metrics.

Azure ML CLI v2 and Python SDK v2 express jobs, components, and pipelines as YAML; components are reusable, versioned steps with typed inputs and outputs.

Model registration gate rule: promote a model only if it beats the current production model on the primary metric and passes fairness and drift checks on a holdout set.

Infrastructure as code: define workspace, compute, datastores, and endpoints in Bicep or Terraform; keep environment definitions in version control so deployments are reproducible.

Trigger design: CI on pull request, CD on merge to main or on model registration event; use separate dev/test/prod workspaces and promote artifacts, not retrain per environment.

Automation of retraining: schedule a pipeline with a time trigger or a data-change trigger; the pipeline must include data validation, training, evaluation, and conditional registration.

GitHub Actions or Azure Pipelines authenticate to Azure ML with a service principal or federated credential (OIDC) stored as a secret reference, not a plaintext secret.

Rollback automation: CD keeps the previous deployment addressable so a failed smoke test triggers an automatic traffic shift back.

Problem framing decision rule: classification when the target is discrete and labeled, regression when continuous, clustering when no labels exist; time-series forecasting requires temporal validation splits, never random splits.

Azure ML workspace is the top-level resource; compute (compute instance for dev, compute cluster for training/batch, inference cluster or managed online endpoint for serving), datastores, and registered assets are children of the workspace.

Managed online endpoints suit low-latency real-time scoring with autoscaling; batch endpoints suit large-file, long-running jobs; Kubernetes online endpoints are chosen when you need full cluster control.

Latency budget rule: real-time endpoints target sub-second per-request scoring; if preprocessing exceeds the SLA, move featurization into the model pipeline or a precompute step.

Design for reproducibility by pinning environment images (conda/pip lock), random seeds, and dataset versions; an MLflow run records parameters, metrics, and artifacts for each experiment.

Cost rule: use low-priority (spot) compute clusters for interruptible training, autoscale-to-zero for dev compute, and separate training from inference compute so idle inference nodes can scale down.

Idempotency requirement: training jobs and deployment scripts must be safe to re-run; use unique run names derived from code hash plus data version.

Capacity planning rule: size inference compute from peak requests per second times average latency, then add headroom; autoscale instance count with a minimum of one for production endpoints.

Data asset types: File dataset (any file path, no schema) vs Table (tabular, schema-enforced, supports MLTable for typed, lazy loading with schema validation).

Data drift vs concept drift: data drift is a change in input distribution P(X); concept drift is a change in the relationship P(Y|X); drift detection on inputs will not catch concept drift alone.

Leakage rule: any feature computed using information unavailable at prediction time (target encoding over the full set, future values, post-event fields) must be removed or computed inside the training fold.

Feature store purpose: single definition of a feature reused across training and inference, with offline store for batch training and online store for low-latency lookup at scoring time.

Time-based features require point-in-time correctness; joins must use the feature value as of the event timestamp, not the latest value, to avoid look-ahead bias.

Missing data decision rule: drop a column when missingness is high and uninformative; impute with median/mode plus a missing-indicator when missingness is informative; never impute using data from the validation or test split.

Cardinality rule: high-cardinality categoricals need hashing, frequency capping, or target encoding computed within folds; one-hot encoding explodes dimensionality and is avoided above a few hundred levels.

Data versioning: register datasets with a version and reference the immutable version in training jobs; never train against a mutable path that changes under the same name.

Hyperparameter search strategies: grid search cost grows multiplicatively with dimensions; random search covers high-dimensional spaces better per trial; Bayesian (TPE) search uses prior trials and is preferred when each trial is expensive.

Hyperdrive in Azure ML defines search space, sampling algorithm (random, grid, Bayesian), primary metric with goal, early-termination policy (bandit, median stopping), and max total/parallel trials.

Early-termination policies stop underperforming runs early: bandit uses a slack factor against the best run; median stopping compares against the running median of the primary metric.

Overfitting signals: training loss keeps falling while validation loss rises, large gap between train and validation metrics, and high variance across cross-validation folds.

Cross-validation rule: use k-fold for i.i.d. data, stratified k-fold for class imbalance, group k-fold when records cluster (patient, user), and time-series split for temporal data.

Regularization levers: L1 (sparsity, feature selection), L2 (shrinkage, weight decay), dropout, early stopping, and reduced model capacity; tune strength on validation, not test.

Imbalanced classification: prefer PR-AUC over ROC-AUC when positives are rare; apply class weights or resampling inside the training fold only, never before the split.

Experiment tracking: log parameters, metrics, and artifacts per run; compare runs in the experiment view; register the best run's model as a versioned asset with tags for the training data version.

Deployment modes: managed online endpoint (real-time, autoscale, key or token auth), batch endpoint (invoke over a data asset, writes scored output), and Kubernetes online endpoint (self-managed cluster).

Blue-green vs canary: blue-green swaps all traffic at once for instant rollback; canary splits a small percentage of traffic to the new deployment and increases gradually.

A deployment is a versioned instance of a model plus environment plus scoring script; an endpoint routes traffic to one or more deployments by percentage, enabling safe rollout.

Scoring script contract: init() loads the model once at startup; run(raw_data) receives JSON and must return JSON-serializable predictions; heavy work belongs in init, not run.

Environment rule: pin base image, Python version, and package versions in the environment; the model and environment are immutable inputs to a deployment, so a change requires a new deployment.

Model formats: MLflow no-code deployment needs an MLmodel file with signature; custom deployments need a scoring script; ONNX is chosen for cross-framework runtime portability.

Auth and networking: endpoints support key-based and Azure AD token auth; private endpoints and virtual network integration are used when traffic must not traverse the public internet.

Rollback rule: keep the previous deployment alive at zero or low traffic so rollback is a traffic-percentage change, not a rebuild.

Model monitoring signals: data drift on inputs, prediction drift on outputs, data quality (nulls, type, range violations), and model performance when ground truth arrives with delay.

Drift metrics: population stability index (PSI) and KL divergence for distribution shift; PSI above roughly 0.2 signals meaningful shift, above 0.25 often triggers retraining.

Performance decay detection requires labeled outcomes; when labels lag, monitor proxy signals (input drift, prediction distribution, feature null rate) and schedule delayed evaluation.

Application Insights integration captures endpoint request logs, latency, error rates, and dependency calls; alerts on latency percentiles (p95, p99) rather than averages.

Retraining triggers: scheduled cadence, drift threshold breach, performance below SLA, or data pipeline schema change; retraining must reuse the same evaluation gates as the original model.

Observability rule: log the model version, deployment name, and request ID with each prediction so incidents can be traced to a specific artifact.

Alert design: alert on symptoms (error rate, latency, drift score) with severity tiers; avoid alerting on every metric to prevent fatigue; include runbook links in alert payloads.

Data collection for monitoring: enable endpoint data collection to a storage target, respecting privacy rules; sample high-volume traffic rather than logging every payload.

Fairness metrics: demographic parity compares positive rates across groups; equalized odds compares true/false positive rates; choose the metric that matches the harm and the legal context, and report disaggregated metrics.

Interpretability tools: global feature importance (permutation, SHAP summary) explains the model overall; local explanations (SHAP force/waterfall, LIME) explain a single prediction; use for debugging and for adverse-action explanations.

Error analysis: slice evaluation by sensitive and business-relevant subgroups; an aggregate metric can hide a subgroup with materially worse performance.

Privacy: differential privacy adds calibrated noise to protect individuals; k-anonymity and generalization reduce re-identification; never train on PII without a lawful basis and minimization.

Security controls: workspace uses Azure RBAC and managed identities; secrets live in Key Vault, never in code or environment files; compute sits in a virtual network with private endpoints.

Model governance artifacts: model card (intended use, limitations, metrics, ethical considerations), datasheet for datasets, and an audit trail of who approved which model version.

Human oversight rule: high-impact decisions require a human-in-the-loop review path and a documented appeal or override mechanism.

Responsible AI checklist gates: fairness testing, interpretability review, privacy impact assessment, and harm/limitation documentation before production approval.

Two pipelines: CI validates code, runs unit tests, and builds the environment image; CD registers the model and deploys to an endpoint, gated by evaluation metrics.

Azure ML CLI v2 and Python SDK v2 express jobs, components, and pipelines as YAML; components are reusable, versioned steps with typed inputs and outputs.

Model registration gate rule: promote a model only if it beats the current production model on the primary metric and passes fairness and drift checks on a holdout set.

Infrastructure as code: define workspace, compute, datastores, and endpoints in Bicep or Terraform; keep environment definitions in version control so deployments are reproducible.

Trigger design: CI on pull request, CD on merge to main or on model registration event; use separate dev/test/prod workspaces and promote artifacts, not retrain per environment.

Automation of retraining: schedule a pipeline with a time trigger or a data-change trigger; the pipeline must include data validation, training, evaluation, and conditional registration.

GitHub Actions or Azure Pipelines authenticate to Azure ML with a service principal or federated credential (OIDC) stored as a secret reference, not a plaintext secret.

Rollback automation: CD keeps the previous deployment addressable so a failed smoke test triggers an automatic traffic shift back.