Model training, tuning, and experimentation
Model training, tuning, and experimentation tests reproducible jobs, reusable components, hyperparameter sweeps, AutoML, tracking, and environment control.
Core rules
- Use command jobs for single scripts and pipeline jobs for multi-step workflows. Put reusable steps in components with typed inputs/outputs.
- Pin the environment: base image, conda/pip dependencies, and code version. Log parameters, metrics, artifacts, and models with MLflow or Azure ML tracking.
- For sweeps, define search space, sampling method (random, grid, Bayesian), primary metric, objective, max trials, and early termination. Bayesian is efficient for expensive trials; grid is exhaustive but costly.
- Select the best model on validation data; evaluate once on a held-out test set and register the chosen model with tags.
Worked example
A team tunes XGBoost for credit default. It uses a sweep with Bayesian sampling, 50 max trials, primary metric AUC, and bandit early termination with slack 0.2. Each trial logs learning_rate, max_depth, and AUC. The best validation AUC is 0.86; the test AUC is 0.84, so the model is registered with both values and the data asset version. The test set was not used for tuning, so the reported test score is not optimistically biased.
Traps
- Tuning on the test set or repeatedly checking it.
- Forgetting max_total_trials or timeout, causing runaway cost.
- Changing dependencies mid-experiment so results are not comparable.
- Registering a model without lineage to job, data, and environment.