Model Development: Reproducible Training, Features, and Tuning on Databricks
The Model Development domain tests whether you can build a model on Databricks that another engineer can reproduce and a serving system can load. Expect scenario questions on MLflow tracking, Feature Store training sets, hyperparameter tuning, Spark ML pipelines, and model packaging. The exam rarely asks for a definition alone; it asks which logged artifact or join key makes a failed model auditable.
Track the run as the unit of reproducibility
An MLflow run records params, metrics, tags, artifacts, and the model. mlflow.autolog() is convenient, but you still log the Delta table version (or time-travel timestamp), the Git commit, the environment, and the random seed. Nested runs keep each tuning trial separate while a parent run aggregates. A model logged without a signature or input example can be registered but is harder to validate at serving time.
Build features without leakage
Feature Store tables in Unity Catalog have a primary key and an optional timestamp key. FeatureLookup names the table, keys, and features; create_training_set joins those features to the label dataframe as of each label's timestamp. That point-in-time join is the core rule: if you join the latest feature value to an old label, you leak future information. Publish to an online store only when real-time lookup is required.
Tune and evaluate deliberately
Hyperopt's fmin with SparkTrials (or Optuna) distributes trials; the objective logs a validation metric and returns it. Keep a held-out test set untouched until the final candidate is chosen. In Spark ML, a Pipeline chains StringIndexer, OneHotEncoder, VectorAssembler, and the estimator; CrossValidator tunes stages. Persist the fitted PipelineModel or log it with the appropriate MLflow flavor.
Worked example
A team trains a loan-default model. The label table has loan_id, application_ts, and default. Features live in ml.features.loan_features with keys loan_id and feature_ts. They create FeatureLookup(table_name="ml.features.loan_features", lookup_key="loan_id", timestamp_lookup_key="application_ts", feature_names=[...]), then create_training_set(label_df, features). After training XGBoost, they call infer_signature(X_train, predictions), log input_example, and tag the Delta version. Reasoning: the timestamp join prevents leakage, the signature lets the endpoint reject malformed payloads, and the data version supports audit.
Traps that cost marks
- Tuning on the test set, then reporting that score as generalization.
- Logging a model without a signature or input example, then blaming serving.
- Joining current feature values to historical labels.
- Assuming a Python UDF scales; prefer pandas UDFs or native Spark functions.
- Forgetting that Feature Store online tables are a separate publish step.