Propensity Versus Uplift in a Retention Program
A subscription business has 500,000 customers and 4.5% monthly churn. A retention offer costs USD 35 per recipient and, on average across past campaigns, lifts retention by 6 percentage points among recipients. The customer-level lifetime margin is USD 600. A propensity model ranks customers by churn probability; the top decile churns at 18% versus 4.5% overall. A separate uplift model, validated on last quarter's randomized holdout, finds a persuadable segment of 40,000 customers with a 9-point incremental retention effect; the remaining customers show near-zero or negative uplift. The CFO wants to mail the top propensity decile because those customers are the most likely to leave.
1. Evaluate the CFO's plan economically and compare it with targeting the persuadable segment. Show the arithmetic.
Read the worked answer
CFO plan: top propensity decile (50,000 customers)
- Baseline churners in the decile: 50,000 × 18% = 9,000.
- Apply the 6-point average lift: 3,000 saved.
- Value: 3,000 × USD 600 = USD 1.80M.
- Cost: 50,000 × USD 35 = USD 1.75M.
- Net: +USD 50,000. Marginally positive, and entirely dependent on the 6-point average holding in this decile - which the uplift model says it does not.
Uplift plan: 40,000 persuadables
- Incremental saves: 40,000 × 9% = 3,600.
- Value: 3,600 × USD 600 = USD 2.16M.
- Cost: 40,000 × USD 35 = USD 1.40M.
- Net: +USD 760,000.
Conclusion Mailing the persuadables produces roughly 15 times the net benefit with 10,000 fewer offers. The propensity plan spends most of its budget on customers who would have stayed anyway or who cannot be moved.
Caveat to state: the 6-point figure is a campaign average, not a decile-specific effect. Applying it to a high-risk decile is exactly the extrapolation the uplift analysis was built to check.
2. Explain to the CFO why the most likely to churn are not the best to target, using the segment structure the uplift model implies.
Read the worked answer
Three kinds of customer the propensity score cannot distinguish
- Sure things - high churn probability, but they will leave regardless of the offer. Mailing them pays USD 35 for nothing.
- Lost causes / sleeping dogs - low or moderate risk, but the offer itself irritates them or reminds them to cancel. Uplift here is zero or negative; mailing them destroys value.
- Persuadables - they would churn without the offer and stay with it. This is the only group where the USD 35 generates a return.
Why propensity fails Propensity answers who will churn. It says nothing about whose behavior the offer changes. A customer at 40% churn risk could be a sure thing, a persuadable, or a sleeping dog; the score is identical. Ranking by risk therefore maximizes the number of offers sent to people the offer cannot influence.
The empirical evidence The randomized holdout is the key asset: the 9-point effect in the persuadable segment versus near-zero or negative elsewhere is a measured causal contrast, not a correlation. That is a stronger basis for targeting than any risk ranking.
One-line summary for the CFO We are not trying to predict who leaves; we are trying to change who leaves. Those are different models, and only the second one tells us where USD 35 does any work.
3. Recommend a targeting and measurement plan for next quarter, including safeguards.
Read the worked answer
Targeting
- Mail the 40,000 persuadables only. Do not mail the sleeping-dog segment even though it may be cheap to reach.
- Order offers within the persuadable segment by expected incremental value (uplift × USD 600 minus USD 35) if budget is capped below 40,000.
- Re-score monthly; the persuadable segment drifts as the customer base and product change.
Safeguards
- Hold a randomized control group (10–20% of the persuadable segment) receiving no offer. Without it, next quarter's uplift cannot be re-estimated and the effect will decay silently.
- Re-validate the uplift model on fresh holdout data before scaling spend; last quarter's validation is one data point.
- Watch for treatment-effect drift - if the holdout shows the incremental effect falling toward 4–5 points, the economics of the 40,000 target set erode quickly at USD 35 per contact.
- Monitor the negative-uplift segment in a small, ethically bounded test if there is doubt about the sleeping-dog finding; do not roll it out broadly.
- Report net value, not response rate. A response rate can rise while net benefit falls if the wrong customers are mailed.
Decision rule to agree up front Continue the program while measured incremental retention in the holdout exceeds the break-even effect, which is USD 35 / USD 600 ≈ 5.8 percentage points. Below that, the offer loses money regardless of how many customers it reaches.