Churn model with training-serving skew
A subscription streaming service has 12 million customers. A churn model is trained on a nightly account snapshot plus 90 days of clickstream events archived from Event Hubs to Parquet. Offline AUC is 0.91, but the online model scores 0.72 on a random 1% holdout. The feature support_tickets_30d is computed in the training notebook by querying the CRM with ticket_date >= current_date - 30; the online scorer computes the same feature in a .NET service using the request timestamp. A second feature, avg_watch_minutes_7d, is backfilled over 90 days using today’s device-to-plan mapping, which overwrites the mapping when customers upgrade. Data scientists rebuild feature logic separately from engineers. The team wants to add 20 features next quarter and cannot reproduce last quarter’s model.
1. Diagnose the likely causes of the offline/online gap and explain how each cause affects the two features.
Read the worked answer
Both features are temporally inconsistent. For support_tickets_30d, training filters ticket_date >= current_date - 30 at notebook run time, while the churn label refers to a later observation window. That can include tickets after the observation date, which is leakage, and makes the training distribution depend on when the notebook ran. Online scoring uses the request timestamp, so the same customer gets a different value. For avg_watch_minutes_7d, backfilling 90 days with today's device-to-plan mapping rewrites history after upgrades; the model learns from mappings that did not exist at event time. Online uses the current mapping, so parity fails again. The gap is therefore not just overfitting: feature definitions, as-of timestamps, and code paths differ. A random holdout also may not reflect the time-based production distribution, so the offline estimate is optimistic.
2. Recommend a feature-management design that supports 20 new features and point-in-time correctness.
Read the worked answer
Introduce a feature store with versioned feature definitions, entity keys such as customer_id, event timestamps, and point-in-time joins. Compute features once in a shared transformation and materialize offline and online from the same definition. For support tickets, join tickets to the observation timestamp and use only tickets before label time; store feature values keyed by customer and as-of time. For device mapping, keep effective-dated rows and join as of event time. Backfills must use as-of joins, not current tables. Add TTL and freshness SLAs, and a registry for the 20 new features with owners. Online serving reads from the online store by entity key; training reads the offline store with point-in-time correctness. This removes duplicated .NET and notebook logic and makes last quarter's model reproducible if the feature versions and data snapshots are retained.
3. Propose validation and monitoring to prevent recurrence.
Read the worked answer
Run an offline-online parity job on the same entity IDs and timestamps; compare feature values and alert on mismatch. Shadow score a production sample and compare it with offline replay. Monitor feature freshness, null rates, and distribution drift. Add label-delay-aware evaluation and time-based backtests rather than only random holdouts. Block model promotion unless parity and feature validation pass. Track feature-version usage per model and retain the training data snapshot. These controls catch both code divergence and temporal leakage before a model reaches production.