Feature Store Leakage in a Demand Forecast
A retail data science team is building a weekly demand forecast for 4,000 stores. During development, validation error looked excellent: MAPE of 9% on a random train-test split. After deployment, live MAPE was 24%. The pipeline pulls features from a shared feature store. One feature, 'store_weekly_returns_ratio', is computed from the same week's sales and returns, and another, 'promo_lift_estimate', is refreshed nightly using the full week's completed data. The random split mixed weeks across train and test, and the team used the default feature-store point-in-time setting of 'latest'. Forecasts are generated Monday morning for the coming week, but some store systems submit corrections through Wednesday. The team wants to know why live performance diverged and how to fix the evaluation and the pipeline.
1. Identify the leakage mechanisms and explain how each inflates validation performance.
Read the worked answer
Two leaks operate. First, 'store_weekly_returns_ratio' is derived from the same week's sales and returns that the model is predicting, so it encodes the target. A random split places some rows from the same week in train and test, letting the model learn week-specific patterns that will not exist at forecast time. Second, 'promo_lift_estimate' is refreshed nightly using the full week's completed data, so when predicting a week, the feature may already contain outcomes from that week. The feature store's 'latest' point-in-time setting retrieves the most recent value regardless of when it was actually available, silently serving future information. Together these make validation optimistic: the model appears to predict with knowledge it will not have Monday morning. The 9%-to-24% gap is consistent with target leakage rather than ordinary overfitting.
2. Describe how you would redesign the validation split and the feature retrieval to reflect the Monday-morning forecast.
Read the worked answer
Replace the random split with a time-based, rolling-origin evaluation: train on weeks up to time t, predict week t+1, roll forward, and never let future weeks enter training. For each prediction, define the as-of timestamp as Monday 00:00 and retrieve only feature values that were materialized before that timestamp. Set the feature store to point-in-time joins keyed to the forecast origin, not 'latest'. Remove or lag 'store_weekly_returns_ratio' so it uses only prior completed weeks; if a same-week signal is genuinely needed, it must come from data available Monday, and its availability must be proven. For 'promo_lift_estimate', require a versioned snapshot with an explicit availability time and verify that nightly refresh timestamps precede the forecast origin. Handle Wednesday corrections by treating them as later-arriving data that cannot be used at prediction time, and record their effect separately.
3. What checks would confirm the fix, and what residual risk remains after remediation?
Read the worked answer
Confirm by recomputing MAPE under rolling-origin evaluation with point-in-time features and comparing to live results; a close match (for example within a few points) suggests leakage is removed. Add a leakage test that, for a sample of predictions, asserts every feature's availability timestamp is before the forecast origin. Run a feature-ablation check: if dropping a suspect feature sharply worsens validation but not live performance, leakage likely remains. Residual risks: late corrections may still create small optimism if any feature aggregates them; point-in-time metadata can be wrong if upstream jobs mislabel timestamps; and rolling-origin evaluation uses fewer training weeks early in the series, so early-fold error may be pessimistic. Keep a live shadow evaluation and alert when validation and live MAPE diverge by more than a set tolerance.