Model Development, Training, and Experiment Tracking
Domain 3 covers experiment design, reproducible training, tracking, and model selection. The exam emphasizes evidence: parameters, metrics, artifacts, environment, and data versions must be traceable to a registered model.
Tracking rules
- Log git commit, dataset hash, feature version, hyperparameters, random seeds, library and driver versions, and hardware.
- Record metrics on a fixed validation split and keep the test set untouched until final approval.
- Use a model registry with stages and immutable artifact versions; a registry entry should link to the run that produced it.
- For HPO, change one factor family at a time and report variance across seeds, not a single lucky run.
Intel acceleration
Intel Extension for PyTorch and Intel-optimized TensorFlow expose CPU BF16/FP32 optimizations; Intel Extension for Scikit-learn accelerates classical estimators. Acceleration changes latency and sometimes numerics, so track both quality and performance in the same experiment.
Worked example
You compare a baseline FP32 PyTorch model with a BF16 run on an Intel CPU. Log the same seed set, data hash, batch size, and validation split. Baseline: accuracy 0.912, training 40 min. BF16: accuracy 0.910, training 26 min. Reasoning: a 0.2-point drop may be within seed variance, so run three seeds; if the confidence interval overlaps and latency improves, BF16 is a candidate, but promote only after the frozen test set and a latency check on the target serving host.
Traps
- Tracking only the best metric and losing the run configuration.
- Changing data, code, and hyperparameters in one comparison.
- Tuning on the test set.
- Ignoring hardware and library versions that change results.