Sourcing Data for an ED-Risk Targeting Program

Outcome variable (ED visit in next 6 months)

• Claims are the only defensible source: they are member-level, dated, and cover the full enrolled population. Use revenue codes plus place of service, and exclude out-of-network claims paid without encounter detail.
• Risk: 8% of claims still pending at 90 days means recent months undercount ED visits. A model trained on immature months learns a systematically lower event rate.

Predictors

• Claims history - strong, member-level, temporally aligned. Primary feature source.
• Portal clickstream - usable but only exists for 41% of members and only after Jan 2023. Absence of a click does not mean absence of engagement; the field is missing-not-at-random.
• ZIP socioeconomic append - coarse and ecological. It predicts at area level, so it can smuggle in neighborhood proxies for race, age, or language and produce inequitable targeting.
• Survey - 9% response, 12,000 members. Self-reported and tiny relative to the population; fine for validating a hypothesis, unusable as a training feature.

Ranked risks

1. Immature outcome labels (bias in the target itself).
2. Portal coverage bias masquerading as a behavioral signal.
3. Ecological predictor driving inequity.
4. Survey nonresponse.

Mechanism in both cases: the missingness is related to the outcome. Members who respond to surveys and members who log into a portal are, on average, healthier, more engaged, and higher-income than non-responders. If engagement features are used as predictors, the model learns relationships that hold only in the engaged subpopulation and then mis-scores everyone else - who are precisely the members most likely to have avoidable ED visits.

Survey (9% response)

• Do not use survey fields as model inputs.
• Instead, use it as a validation sample and reweight it to known population margins (age, sex, plan, region) before drawing any inference.
• Compare survey responders to non-responders on claims-derived characteristics to quantify how different they are.

Portal (41% coverage)

• Create an explicit portal_available flag and test whether the model's performance is stable across it.
• Use only within-member behavior where possible (change in clicks, not raw counts).
• Run the model with and without engagement features and report the delta in lift. If lift depends on them, the model is partly a proxy for digital literacy, not risk.

Test for both: score the top 5% among covered and uncovered members separately; if the flagged populations look different in ways not explained by clinical risk, the bias is real.

The problem

• A 60–90 day lag plus 8% still pending at 90 days means the most recent two to three months are incomplete. Training on them undercounts ED visits and biases predictions downward for exactly the members whose behavior you most need to catch.
• Scoring today on features from an immature window produces a ranking that is partly a measure of billing processing speed.

Design changes

1. Truncate the label window. Define the outcome as ED visits observed through a cutoff at least 120 days back, so pending claims have time to adjudicate. Drop the immature tail rather than imputing it.
2. Freeze a validation quarter. Hold out the last fully mature quarter as a locked test set; never train on it.
3. Snapshot the features as of the scoring date. No post-outcome leakage: predictors must be knowable at the moment outreach would begin.
4. Refresh on a rolling 24-month window so seasonality is represented without letting staleness dominate.
5. Reconcile the 8% pending with a run-out factor by claim type, and document the assumption; if the run-out varies by region, apply it by region.
6. Align output to capacity: rank and cut at 4,000 per quarter, but report the score distribution so the team can see how sharply risk falls off at the cut point - if it is flat around 4,000, the cutoff is arbitrary and should be revisited with clinical stakeholders.