A renewal dataset with late features and duplicated policies
A team builds a renewal prediction table scored at the beginning of each calendar year. The source contains policy-period rows, claim records, and multiple payment transactions per claim. Some events occurred in December but were first entered into the system in February. A proposed predictor totals the following March's payments because those figures appear in the latest extract. The analyst also imputes missing values using the entire assembled dataset and target-encodes categories before splitting. Several customers have repeated policy years, while the stated business evaluation concerns performance on new customers. The merged row count has increased, but no one has reconciled total exposure. The team needs to reconstruct what could genuinely have been known and decide which validation population the analysis represents.
1. Identify the main leakage and observation-unit problems.
Read the worked answer
Future payments and late-entered records cannot simply be treated as available at the renewal timestamp. Joining claim and payment rows can duplicate policy-period exposure. Full-data imputation and target encoding can also leak evaluation information. Define the scoring-time view and reconcile identifiers, row units, counts, exposure, and losses.
2. Propose an appropriate pipeline and evaluation design.
Read the worked answer
Use an availability-aware feature cutoff and documented joins, then separate customers for the new-customer objective while respecting relevant time structure. Fit imputation, encoding, and selection inside training folds, using out-of-fold training target encodings where appropriate. Apply the learned schema unchanged to the held-out sample.