Warranty Propensity: 0.94 in Validation, 0.71 in Production
Situation. A warranty-claim propensity model scores every consumer laptop at ship time so the retention team can send a setup-and-care email. Training data: 1.2M units shipped over 26 months, 2.8% filed a claim within 90 days. Validation AUC was 0.94; the live pilot on 40,000 units scored 0.71.
The feature audit found: tickets_last_30d (a support ticket exists only if the customer already contacted support), parts_shipped_flag (a replacement part was already dispatched), and ship_month used as a random split key, so units from the same month appear in both train and test. Claims are snapshotted nightly, and labels are truncated at 60 days for units shipped in the most recent month. The retention team can send 1,500 emails a week.
1. Explain the 0.94 to 0.71 gap. Which features are invalid and why, and what validation scheme should have been used?
Read the worked answer
The gap is leakage, and it is in the features first.
tickets_last_30d and parts_shipped_flag are post-outcome features. A customer has a support ticket or a dispatched part because they already started the claim process. At training time these correlate with the label at 0.9+, so the model reads them as near-certain evidence. At scoring time - ship day - both are constant zero (or NULL) for every unit, so the model loses its strongest inputs and falls back to weak ones. tickets_last_30d may also leak across time even when computed as intended: a ticket on day 45 for a unit shipped in March puts future information next to a March prediction.
The split compounds it. A random row split on 26 months of shipments puts the same month's units on both sides. Any month-level condition - a firmware release, a promotion, a supplier change, a claims-processing backlog - is learned in train and rewarded in test. The model memorises month, not risk.
Correct scheme: (1) drop every feature whose timestamp can fall after the prediction moment, and enforce it with a feature-availability contract rather than a code review; (2) split by ship month - train on months 1–22, validate on 23–24, test on 25–26; (3) validate on units whose labels are fully mature (see next task); (4) prefer AUC computed on genuinely out-of-time data, and add calibration (predicted vs observed claim rate by decile) because the threshold decision needs probabilities, not ranks.
2. Redefine the target and observation window so labels are mature, and say what to do with the most recent month of shipments.
Read the worked answer
The target must be stated as a triple: event, window, as-of date.
Event: a warranty claim filed and accepted (not merely opened - opened claims get rejected or withdrawn, and mixing them adds label noise that the model will happily fit). Window: 90 days from ship date, because that is the window the business cares about and the one the intervention can affect. As-of: the label for a unit is mature only on ship_date + 90 days + processing lag. With a nightly snapshot truncated at 60 days, units shipped in the last ~30 days have systematically missing positive labels - they look like non-claimers. Training on them teaches the model that recent units are low risk, which is precisely the population you are scoring.
What to do with the latest month: exclude it from training and validation entirely until its labels mature, and add a label_mature flag with a scheduled retraining cut so exclusion is automatic rather than remembered. For monitoring, score it (that is the job) but never evaluate on it. If data volume is tight, a survival-style correction is possible - treat immature units as censored at their observed follow-up and model time-to-claim - but that is a second project, not a hotfix. For the pilot review, hold out a mature 40,000-unit cohort and compare against that, not against the truncated training sample.
3. Each email costs $18 all-in; a prevented claim saves $310; the email is assumed to prevent 25% of claims among customers who would have claimed. At what precision does outreach break even, and does the 1,500-per-week cap help or hurt?
Read the worked answer
Break-even precision. Expected benefit per email = p x 0.25 x 310 = 77.5p, where p is the claim probability of the person emailed. Setting that equal to $18 gives p = 18/77.5 = 23.2%. Against a 2.8% base rate, the model must deliver roughly an 8.3x lift in the contacted group - a very high bar for a 90-day laptop claim.
Implication. If the model's top 1,500 per week averages 4x lift (p ≈ 11.2%), expected benefit is $8.68 against $18 cost: each email destroys about $9.30. A capacity cap does not fix that; it only limits the loss. Volume is not the lever - precision is.
What to actually do: (1) rank by expected value (score x 77.5 − 18) rather than by score, and contact only the positive-value tail, which may be far below 1,500; (2) treat 1,500 as a ceiling, not a target - an unfilled queue is a valid outcome and should be reported as such; (3) challenge the two assumption numbers, because 25% efficacy is doing enormous work and is untested; (4) run a randomised holdout of about 20,000 contacted customers, because the only way to defend the $77.5 figure is to measure prevented claims, not predicted ones.