Offline AUC 0.89, Online AUC 0.72
A churn model uses a 2B-row event table and a daily feature table ml.churn_features. Training data is built with a hand-written AS OF join on event_date; batch scoring uses a separate SQL query that joins each customer to the current feature row. Both paths compute days_since_last_order and fill nulls, but training defaults null tenure to 0 and scoring defaults to -1; training also uses UTC and scoring uses local time. Offline validation AUC is 0.89, but online AUC over two weeks is 0.72. The label is 30-day churn. The team wants to keep using the existing feature table.
1. Identify the most likely causes of the offline-online gap and describe how to restructure the pipeline to eliminate them.
Read the worked answer
The gap is consistent with training-serving skew, not just model drift. Three concrete causes are visible: the two paths implement feature logic separately, null handling differs (0 vs -1), and timezone handling differs. A fourth, larger risk is point-in-time correctness: if the hand-written AS OF join accidentally uses feature values from after the label window, offline AUC is inflated.
Restructure around one feature definition. In Unity Catalog Feature Engineering, define the feature logic once and build training sets with FeatureLookup and a timestamp_lookup_key so each label row gets the feature value as of the label event time. Log the resulting training set with the model. At serving time, use the same feature definitions through the online store or the same lookup path, not a hand-written query. Replace hard-coded null defaults with a shared imputation rule inside the feature definition, and standardize all event timestamps to UTC.
2. Design a test that would catch this class of bug before deployment, using Databricks features.
Read the worked answer
Build a parity test on a fixed sample of customer IDs and event timestamps. Compute each feature through the offline training path and through the serving path, then assert exact or tolerance-based equality; a null-tenure mismatch of 0 vs -1 would fail immediately. Run this test in CI against a small Delta fixture.
In addition, log the training set as an MLflow artifact so the exact feature values used for training are auditable. Use mlflow.evaluate on a time-based holdout rather than a random split, and compare offline metrics to a backfill scored with the serving path. After deployment, enable inference tables on the serving endpoint, join predictions to delayed 30-day labels, and compare online AUC and feature distributions to the offline baseline. This turns the offline-online gap into a detectable regression rather than a post-hoc surprise.