Feature Store Latency vs. Training Throughput

The gap is almost certainly training-serving skew on the 30 streaming-derived features, not model capacity. Adding trees optimizes the wrong thing: if the offline feature values differ systematically from what the online path emits at request time, a stronger model just fits the skewed distribution more tightly.

The mechanism: the nightly snapshot aggregates event data after a batch cutoff, so late-arriving events (up to 6 hours) are either dropped or attributed to the wrong window. The online path sees events as they arrive, so a user's "events in last 24h" at scoring time includes traffic the snapshot never captured. The two distributions differ most for high-activity users, which is exactly where churn signal concentrates.

Confirmation plan, cheapest first:

1. Log the exact feature vector used at online scoring for a sample of requests, and join it back to the offline snapshot for the same entity and timestamp. Compute per-feature distribution drift (PSI or KS) across the 30 streaming features versus the 90 batch features. Expect drift concentrated in the streaming set.
2. Shadow-score offline: replay the logged online vectors through the current model. If AUC recovers toward 0.91, the model is fine and the features are the problem.
3. Compare point-in-time correctness: for a sample of entities, recompute the 30 features as of the scoring timestamp from raw events, and check whether the snapshot value matches. Mismatch rate quantifies the skew.

Only after the feature parity check passes should anyone discuss model changes. If parity holds and the gap remains, then look at label leakage in the snapshot or population differences between the holdout and live traffic.

Two problems, one design: move to a point-in-time-correct feature store with incremental materialization.

Fixing skew:

• Define each streaming feature once, in a shared transformation library, and use it in both paths. The offline path replays the same logic over event history; the online path runs it over the live stream. One definition, two executors.
• Make offline reads point-in-time joins: for each training row, fetch feature values as of the label timestamp, including late-arriving events up to the watermark. This is what removes the 6-hour attribution error.
• Store streaming features in an online store keyed by entity with event-time versioning, and backfill the offline store from the same changelog so both derive from one source of truth.
• Add a parity test in CI: for a fixed sample of entities and timestamps, compute features via the offline and online code paths and assert equality within tolerance. Fail the build on divergence.

Shortening the build:

• The 9 hours is likely full recomputation of 120 features over all history. Switch to incremental materialization: recompute only partitions touched since the last run, using the event watermark. Most days only a slice of history changes.
• Precompute the 90 batch features on a slower cadence (weekly or on-change) and only refresh the 30 streaming features nightly; the streaming set is the volatile one.
• Partition by event date and entity hash so the incremental job is embarrassingly parallel and does not rescan cold partitions.

Cost: incremental jobs run more often but touch far less data, so total compute typically drops. The online store adds storage and serving cost, offset by not maintaining two divergent pipelines. Reuse existing Kafka and lake infrastructure rather than standing up a new stack.

Matching AUC means the pipeline is consistent; it does not mean the decision threshold matches the business cost structure. The complaint is about decisions, not discrimination.

Check, in order:

1. Threshold provenance. Was the operating threshold chosen on the old 0.91 model and carried over? A recalibrated model with the same AUC can shift the score distribution. Recompute the threshold on current holdout data at the target precision/recall.
2. Calibration. AUC is rank-based and blind to calibration. If predicted probabilities are inflated, a fixed 0.5 cutoff flags far more people. Plot a reliability curve; if miscalibrated, apply isotonic or Platt scaling fit on a recent holdout.
3. Class and segment balance. Churn base rate drifts seasonally. If the live churn rate fell from, say, 8% to 5%, a threshold tuned at 8% over-flags. Evaluate precision per segment (tenure, contract type) - loyal long-tenure customers may sit in a region where the model is systematically over-confident.
4. Label definition drift. Did "churn" change definition (e.g., 60-day vs 90-day inactivity)? A stale label makes the model target the wrong event.
5. Feedback loop. If retention offers were sent to flagged users, their behavior changed; the model may be learning from its own interventions.

Change: set the threshold from an explicit cost matrix (cost of a wasted offer vs. cost of a lost customer), recalibrate probabilities, and add segment-level precision monitoring with alerts. Consider a two-stage decision: high-precision auto-offer, mid-band routed to human review. Do not retune the model architecture to fix a threshold problem.