I85PREP
⌘ K
DE MLO study materials
DE MLO study resources

DE MLO: Model Development, Training, and Tuning

DE MLO · Work through your exam, one topic at a time.

All guide sections

Model Development, Training, and Tuning

What the exam tests

This domain tests how you build reproducible training workflows on Databricks: Spark ML pipelines, hyperparameter tuning, and Feature Store integration. Expect questions on avoiding leakage, choosing validation strategies, and logging the tuning process so the best model can be reproduced.

Core rules

  • Spark ML uses estimators and transformers. A Pipeline chains stages such as StringIndexer, OneHotEncoder, VectorAssembler, and a classifier. Fit the entire pipeline on training data only so preprocessing is learned without validation or test leakage.
  • Use CrossValidator for k-fold tuning or TrainValidationSplit for a single split. Build grids with ParamGridBuilder; evaluate with an explicit evaluator.
  • Hyperopt uses fmin with a search space and an objective. SparkTrials distributes trials across a cluster and integrates with MLflow autologging.
  • Databricks Feature Store uses FeatureLookup, create_training_set, and train_model. Feature lookups are point-in-time correct when feature tables have timestamp keys, which prevents future information from entering training.
  • Split temporal data by time, not randomly. Random splits leak future patterns into training for forecasting and fraud use cases.

Worked example

A team predicts churn from customer features. They create a training set with FeatureLookup using a label table and feature tables keyed by customer_id with event_ts. They split by event_ts: older data trains, newer data validates. The pipeline includes categorical encoding and a random forest. Hyperopt tunes maxDepth and numTrees with SparkTrials, and each trial logs validation AUC. The best trial is refit on train plus validation, then evaluated once on the held-out test period. Reasoning: the time split mirrors deployment, the pipeline prevents preprocessing leakage, and the single test evaluation keeps the estimate honest.

Traps that cost marks

  • Fitting an encoder or scaler on the full dataset before splitting. This leaks validation information into training.
  • Randomly splitting time-series data, which lets future data train the model.
  • Tuning on the test set or repeatedly checking test performance during search.
  • Using Feature Store lookups without timestamp logic and assuming point-in-time correctness.
  • Logging only the best score and not the search space, trial params, or data version, making the result hard to reproduce.

Key points

  • Fit Spark ML pipelines on training data only; keep preprocessing inside the pipeline.
  • Use CrossValidator or TrainValidationSplit with explicit evaluators.
  • Hyperopt with SparkTrials distributes trials and logs them to MLflow.
  • Feature Store lookups must be point-in-time correct to avoid leakage.
  • Temporal problems need time-based splits, not random splits.