Late-Delivery Model With Post-Outcome Features
A logistics team predicts late delivery for 300,000 shipments. The label is late (18%). They build a Visual Analysis with a random 70/30 train/test split. Test AUC is 0.94, so they deploy an API endpoint. In production, AUC falls to 0.62 and the late rate is underpredicted. The feature list includes promised_date, carrier, distance_km, customer_zip, delivery_scan_count, last_status_update_hours, and final_route_code. The last three are written by the scanning system after the parcel is delivered or after the promised date. The team retrains weekly on the latest 12 months. A product manager wants to add 'refund_requested' because it is highly correlated with late delivery. You are asked to diagnose the gap and redesign the validation and feature review. No time-based holdout was used.
1. Why does test AUC 0.94 collapse to 0.62 in production? Be specific about leakage and validation.
Read the worked answer
The random split lets rows from the same period appear in train and test, and three features are recorded after the outcome. delivery_scan_count, last_status_update_hours, and final_route_code encode what happened after delivery or after the promised date; they are not available at prediction time. A random split can also let the model memorize period-specific carrier and route patterns. In production, those post-outcome fields are missing or stale, so the model loses its strongest signals. The 0.94 is therefore an optimistic leakage estimate, not evidence that the model generalizes. A time-based holdout would expose the drop earlier. The 'refund_requested' feature is worse: it is a downstream consequence of late delivery and would create target leakage.
2. Redesign the validation and feature review. What would you change in Dataiku?
Read the worked answer
Use a time-based split: train on the oldest 9 months, validate on the next 2, and test on the most recent month. Ensure no shipment from a later period appears in training. In Visual Analysis, set the split policy to time-based using ship_date, or build separate datasets. Remove all post-outcome fields and any field whose value is written after the prediction timestamp; document the timestamp for each feature. For 'refund_requested', exclude it. Review feature importance after retraining; if delivery_scan_count still dominates, the timestamp audit is incomplete. Compare AUC, precision at the late-rate threshold, and calibration by carrier and route. Add a drift check on feature distributions between train and production.
3. What deployment and retraining guardrails would you add?
Read the worked answer
Deploy behind a monitored API with a schema that rejects missing or late-arriving features. Log predictions and actual late outcomes, and compute AUC and calibration weekly. Set alerts for data drift on carrier mix, distance, and scan counts, and for concept drift when the late rate shifts. Retrain on a rolling window with the same time-based validation; do not retrain on rows whose features were captured after the outcome. Use a scenario to automate retraining only after validation passes a minimum AUC and calibration threshold. Keep a champion/challenger comparison before promoting a new model. Also cap the impact of the model on operations until it has passed a shadow period.