Churn Model Request at a Meal-Kit Subscription Service

The request conflates a tool with a decision. A propensity model ranks who might leave; it does not tell you whether an offer changes behavior, and it may be the wrong first move when a controllable operational driver is already visible. I would reframe around the decision: reduce preventable churn caused by late deliveries in the six affected ZIPs, and target retention spend where it produces incremental retention.

That means confirming with the VP: what action follows a high score, who acts, what the offer costs, and what outcome within what horizon counts as success. If the only action is a $22 discount, the analysis should estimate uplift, not just risk. I would also confirm that operations can fix delivery performance; if late deliveries explain a large share of churn, an ops fix may dominate any model.

I would propose a phased statement: first quantify how much churn is attributable to late delivery and estimate the value of fixing it; second build a model to prioritize calls or offers for the residual risk. The first deliverable is a one-page decision memo, not a model, and it should force an explicit choice among operational remediation, targeted retention, or both.

Target: voluntary cancellation within 60 days of an observation date. Exclude involuntary churn (failed payments, expired cards) or model it separately, because a retention offer cannot address it. Population: active subscribers at the observation date with at least one delivered box; new sign-ups a few days into tenure have different dynamics and should be modeled separately.

Pitfalls:

• Leakage. Support tickets or complaints logged after the observation date must not become features. Build a feature cut-off timestamp and enforce it.
• Censoring. Subscribers who have not yet reached 60 days cannot be labeled as non-churners; exclude them or use survival methods.
• Class imbalance. Monthly churn of 7.2% implies a 60-day rate near 14%, so accuracy is misleading. Use precision-recall, lift, and calibration.
• Confounding. The 3.1x late-delivery association may partly reflect ZIP-level income or competition. Stratify or control for those factors before calling it causal.
• Label heterogeneity. Without cancellation reasons, the target mixes price, food quality, and logistics churn, which pulls the model in different directions.

Success metric: incremental retained margin per 1,000 contacted subscribers, net of offer cost and agent time, estimated from a randomized holdout. AUC is a secondary diagnostic; a model with excellent AUC can still lose money if it targets people who would have stayed anyway.

I would accept the deadline only as a scope negotiation, not as a commitment to the full model. Five weeks with two analysts can deliver: a clean target definition and cohort table (week 1); a baseline logistic regression using tenure, plan, delivery lateness, and support contacts (weeks 2–3); a backtest on a held-out period (week 4); and a small randomized pilot design for the discount (week 5). That gives the VP something usable before Q4 and preserves the ability to learn.

I would be explicit about what is not deliverable: causal estimates of the discount's effect, a production scoring pipeline, and reliable performance for new subscribers with little history. I would also ask for a named decision owner and an operations partner, because the late-delivery finding may change the intervention.

If the VP insists on a full production model in five weeks, I would document the risk: no time for leakage checks, no holdout, no privacy or fair-treatment review, and a model that may optimize the wrong action. The compromise is a pilot with a randomized holdout, plus a commitment to a full model only after the pilot shows incremental lift.