Fitline wants a 14-day upgrade prediction in three weeks
Fitline is a fitness app with 900,000 monthly active users, 210,000 paying members, and 14 months of history in mParticle. It wants a predictive audience of users likely to upgrade to the paid annual plan within 14 days, feeding a push campaign and a paid-social lookalike. Conversion to paid is 3.4% per 30-day window. Two constraints: the company migrated off a legacy SDK 11 weeks ago and renamed the core workout events in the process, and iOS push opt-in sits at 31%. The growth team wants to launch in three weeks and has already promised the CFO a 20% lift in upgrades. Past campaigns targeted the top decile of engagement, so historical upgrade rates are contaminated by prior targeting.
1. Is a predictive audience viable on this timeline? Justify your answer against what the model actually needs, and propose what to ship in three weeks instead.
Read the worked answer
Not as specified. Work through what a supervised model needs:
- Label stability. The outcome is 'upgraded within 14 days'. Eleven weeks of clean data gives about five non-overlapping 14-day windows, and the event rename means pre-migration features are not comparable. A model trained now learns the migration.
- Positive volume. 3.4% of 900k MAU is plenty of positives in aggregate, so volume is not the blocker - recency and contamination are. Worth stating explicitly, because the team will assume the problem is data quantity.
- Contamination. Prior campaigns targeted the top engagement decile, so historical labels encode 'was targeted and upgraded', not 'would have upgraded'. Train on that and you rebuild the old targeting rule with extra steps and a confidence score attached.
- The 20% promise. A first model with no holdout is how prediction programmes get cancelled after one quarter.
Three-week plan: (1) fix the taxonomy and backfill the renamed workout events from the warehouse so features are continuous; (2) ship a rules-based real-time audience now - trial users with 2+ sessions and declining workout frequency over 10 days - which is explainable to finance; (3) register the predictive audience with a holdout, a baseline and a defined refresh cadence before the first send.
2. Describe the validation and monitoring design you would insist on before the predictive audience is allowed to drive spend.
Read the worked answer
Set this up before launch, not after:
- Holdout. 10–20% random holdout excluded from every channel in the campaign, not just from the model's send. Lift = treated conversion minus holdout conversion. Raw conversion in the targeted group proves nothing when the group was selected for being likely to convert anyway.
- Baseline. Freeze the 'no campaign' control definition and evaluate over the identical 14-day window. Fitness seasonality is enormous - January rewrites every result - so compare like week against like week.
- Label timing. The outcome window must close before each model refresh. Training on partially observed outcomes makes the newest cohort look systematically worse and triggers a spurious rollback.
- Feedback loop. If targeting changes who upgrades, next quarter's training data reflects the campaign rather than propensity. Mitigate by training on the holdout population where volume allows, keeping campaign-exposure signals (push opens, ad clicks) out of the feature set, and logging audience membership at send time so exposure can be reconstructed.
- Monitoring. Track positive rate, calibration by score decile, and audience size drift. A sudden jump in membership is almost always a tracking change, not a behaviour change - investigate before spending against it.