Model Development and Experimentation
What the domain tests
This domain assesses reproducible model development in Cloudera AI: experiment tracking, parameter and metric logging, artifact capture, model comparison, and registration. Expect questions on experiment runs, random seeds, train, validation, and test splits, cross-validation, leakage, and the handoff from a notebook to a registered model version.
Core rules
- Every experiment should capture code version, data version, parameters, metrics, and artifacts. A metric without its data snapshot is not reproducible.
- Split data before any preprocessing that learns from data. Fit scalers, encoders, and imputers on the training fold only; otherwise leakage inflates validation scores.
- Use a held-out test set once, at the end. Repeatedly tuning on the test set turns it into a validation set.
- Log the model plus preprocessing. A registered model that expects raw fields but was trained on scaled features will fail or silently mispredict in serving.
- Set random seeds for sampling, splitting, and model initialization when comparing runs.
- Compare runs on business-relevant metrics, not only accuracy; for imbalanced classes, inspect precision, recall, F1, and the confusion matrix.
Worked example
A data scientist trains a random forest to predict churn. They log max_depth and n_estimators, record AUC and recall at a fixed threshold, save the fitted pipeline containing imputer, scaler, and model, and register the best run as model version 3. Reasoning: the pipeline artifact ensures serving applies the same transformations; logging recall matters because the business cares about catching churners, not just overall accuracy. The test set is touched only after model selection.
Traps
- Scaling the full dataset before splitting, leaking validation information.
- Registering a bare estimator without its preprocessing steps.
- Logging only the final metric and not the parameters or data version.
- Selecting a model on test-set performance.
- Comparing runs with different seeds or splits as if they were equivalent.
- Ignoring class imbalance and reporting accuracy alone.