DE MLO study plan

Set up a local MLflow tracking server with a file-based backend and log at least three training runs from a small tabular model, capturing params, metrics, and artifacts.

Compare and contrast the roles of runs, experiments, and registered models, and write a one-page summary explaining when each abstraction is appropriate.

Log a dataset artifact and a model artifact separately so you can articulate the difference between data lineage and model lineage.

Tag and annotate runs with custom metadata (git commit, data version, owner) and verify the tags are queryable through the tracking API.

Register a model into a model registry, promote it through at least two stages (or aliases), and record the transition rationale.

Implement a reproducible training entry point that pins dependency versions and seeds, then rerun it to confirm identical metrics within tolerance.

Capture environment reproducibility by exporting the exact package set used for a run and storing it alongside the model artifact.

Document the lineage chain from raw data version to registered model version in a short README.

Build a training pipeline with explicit train/validation/test splits and a fixed random seed, and log the split hashes for auditability.

Run a hyperparameter sweep of at least 12 configurations and log every trial as a tracked run rather than only the best one.

Diagnose overfitting versus underfitting by comparing learning curves on train and validation sets, and write down the evidence for your conclusion.

Evaluate the final model with at least three metrics appropriate to the task and justify why each metric was chosen.

Introduce a deliberate data-quality defect (duplicate rows or label noise) and measure its effect on validation metrics.

Implement feature preprocessing inside the pipeline so that it is fit only on training data, and verify no leakage occurs.

Add a data validation step that fails the pipeline on schema drift or out-of-range values.

Write a model card summarizing intended use, training data, metrics, and known limitations.

Package the trained model behind a REST inference endpoint and verify it returns correct predictions for known inputs.

Implement input schema validation at the serving boundary and return clear errors for malformed payloads.

Measure and record p50 and p95 latency plus throughput for the endpoint under a modest concurrent load.

Describe at least two deployment strategies (for example shadow versus canary) and state which risks each mitigates.

Containerize the inference service and confirm it starts reproducibly from a clean image build.

Define a rollback procedure for a bad model version and rehearse it once end to end.

Instrument the service to emit prediction logs including inputs, outputs, model version, and timestamp.

Set up a baseline dashboard or report that shows request volume, error rate, and latency over time.

Compute drift statistics comparing live prediction inputs against the training distribution and set a threshold that would trigger review.

Design a delayed-label evaluation job that recomputes model quality once ground truth arrives.

Define alerting rules for data drift, prediction drift, and performance degradation, and explain the tradeoff between sensitivity and alert fatigue.

Document a retraining trigger policy that specifies which signals justify retraining and who approves it.

Audit the full lifecycle from data version to deployed model and identify any missing reproducibility or governance evidence.

Draft a governance checklist covering access control, model approval, audit logging, and retention of experiment metadata.

Re-run the end-to-end pipeline from a clean environment to confirm the documented process actually reproduces the deployed model.

Complete a timed mixed-domain practice set and review every missed item against the corresponding domain notes.