AZ MLO cheat sheet

Workspace: an Azure Machine Learning workspace is the top-level resource; it contains compute, datastores, datasets, models, endpoints, and experiments.

Compute choice: CPU clusters for tabular data, GPU clusters (NC/ND series) for deep learning; use low-priority nodes for fault-tolerant training to reduce cost.

Datastores: register Azure Blob, Data Lake Gen2, SQL, or file share connections once; reference them by name in code instead of embedding credentials.

Data assets: version data with a name and version; use URI (file/folder) or MLTable (schema-aware) types; MLTable supports delimited files and Parquet.

Environment: define a curated or custom environment (Docker image + conda YAML) and register it; pin package versions for reproducibility.

Compute targets: use serverless compute for ad-hoc jobs, compute clusters for scalable training, compute instances for interactive dev.

Identity: use workspace managed identity or user-assigned identity to access storage; avoid key-based auth in production; use Azure Key Vault for secrets.

Job types: command job (script), sweep job (hyperparameter tuning), pipeline job (multi-step orchestration), AutoML job.

Data leakage prevention: split train/validation/test before any preprocessing that learns parameters (scaling, imputation); use pipelines to enforce order.

Data exploration: use pandas in a notebook or a data wrangling step; compute summary statistics, missing-value counts, and class balance before modeling.

Feature engineering: use MLTable or a custom transformer; for categorical data use one-hot encoding or target encoding; for text use tokenization and embeddings.

AutoML: configure task (classification/regression/forecasting), primary metric, timeout, and cross-validation folds; AutoML handles featurization and algorithm selection.

Hyperparameter tuning: use Sweep job with search space (choice, uniform, loguniform), sampling (random, grid, Bayesian), and early termination (Bandit, Median stopping).

Training script: use MLflow logging (mlflow.log_metric, log_artifact) or Azure ML SDK logging; log metrics, parameters, and model artifacts for traceability.

Model evaluation: use appropriate metrics (accuracy, F1, AUC for classification; RMSE, MAE, R2 for regression); use cross-validation for small datasets.

Responsible AI: use Fairlearn or Azure ML fairness assessment to measure disparity across sensitive groups; use error analysis to find cohorts with high error.

Experiment tracking: every job run is logged in the experiment; compare runs in studio or via SDK to select the best model.

Compute scaling: for large data use distributed training (PyTorch DDP, Horovod, MPI); set instance count and process count per node.

Model registration: register a model with name, version, and path; registration enables deployment and lineage tracking.

Endpoint types: online endpoint (real-time, low latency, autoscaling) and batch endpoint (large-scale asynchronous scoring).

Online deployment: specify model, environment, scoring script (init and run), instance type, and instance count; set traffic allocation across deployments.

Batch deployment: specify model, environment, scoring script, and compute cluster; invoke with a data asset input and output location.

Scoring script: init() loads model once; run() receives raw JSON and returns predictions; must handle errors and return HTTP status codes.

Autoscaling: configure min/max instances and scale based on CPU, memory, or custom metrics; use for variable traffic.

CI/CD: use Azure DevOps or GitHub Actions with Azure ML CLI (v2) to automate model registration and endpoint deployment; use YAML job definitions.

Security: deploy endpoints with key or token auth; use private endpoints and managed identity for production; disable local auth where possible.

Experiment tracking: all runs are recorded under an experiment; use tags, properties, and metrics to filter and compare runs.

Model monitoring: enable data drift and prediction drift detection on online endpoints; set alert thresholds and schedule monitoring jobs.

Data drift: compare production data distribution to a reference dataset using metrics like population stability index (PSI) or Jensen-Shannon distance.

Model performance monitoring: log predictions and ground truth (when available) to compute accuracy, AUC, or RMSE over time; trigger retraining when thresholds are breached.

Logging: use Azure Monitor and Application Insights for endpoint telemetry (requests, latency, errors); use MLflow for model metrics.

Retraining: automate retraining pipelines triggered by drift alerts or schedules; register new model versions and compare before promoting.

Model registry: use versioning and tags (e.g., staging, production) to manage lifecycle; archive old versions to reduce clutter.

Responsible monitoring: track fairness metrics and explainability outputs over time to detect degradation across cohorts.

Fairness assessment: use Fairlearn metrics (demographic parity difference, equalized odds difference) to quantify disparity; mitigate with reweighting or threshold optimization.

Interpretability: use SHAP (global and local) and model-specific feature importance; log explanations with the model for auditability.

Error analysis: use the Responsible AI dashboard to identify cohorts with high error rates and guide data collection or model improvements.

Privacy: use differential privacy (e.g., SmartNoise) or data aggregation; avoid logging PII; use Azure ML data assets with access controls.

Transparency: document model purpose, training data, evaluation metrics, and limitations in a model card; store with the registered model.

Human oversight: for high-stakes decisions, implement human-in-the-loop review and provide explanations to end users.

Bias mitigation: pre-processing (resampling), in-processing (constrained optimization), post-processing (threshold adjustment) - choose based on stage and trade-offs.

Compliance: align with Microsoft Responsible AI principles (fairness, reliability, privacy, inclusiveness, transparency, accountability); use Azure ML's built-in Responsible AI tools.