Offline-Online Skew in a Streaming Fraud Feature Pipeline
A payments company runs a fraud model on Kafka events. Offline training AUC is 0.91; the online champion scores 0.78 over the same week. Features come from a Feast feature store: batch features are computed nightly in Spark and served from Redis; streaming aggregates are computed in Flink. The event schema includes event_time and ingest_time. Late events arrive up to 48 hours; about 1.2% of transactions are late. The team recently added a transaction_count_24h feature using processing time instead of event time. Training snapshots use the latest feature values, not as-of event time. The serving layer also applies a 24-hour tumbling window anchored at midnight UTC, while training used a rolling 24-hour window per event. Traffic is 600 events/second at peak.
1. Diagnose the most likely causes of the 0.91 to 0.78 offline-online gap and rank them.
Read the worked answer
Rank by likelihood and by whether each cause can explain a 13-point AUC gap. First is point-in-time leakage in training: snapshots use latest feature values rather than values as of event time, so the model learns from future aggregates. Second is the processing-time versus event-time mismatch in transaction_count_24h, which changes the feature distribution at serving. Third is the window definition: training uses a rolling 24-hour window per event, serving uses a midnight-anchored tumbling window, so the same feature means different things. Fourth is late events: 1.2% late by up to 48 hours will shift streaming aggregates, especially for volatile merchants. Fifth is label leakage or a different label definition, less likely but worth checking. The gap is too large for ordinary sampling noise. Confirm with a three-way comparison: log online features, replay them through the offline pipeline, and compare distributions by merchant segment; then retrain with as-of joins and see whether the gap closes.
2. Design a remediation and validation plan before the next retrain.
Read the worked answer
Fix the semantics before adding infrastructure. Rewrite the batch feature jobs to compute as-of event_time joins, replace processing time with event_time in the streaming job, and make the training and serving window definitions identical, ideally with a shared feature definition library. Backfill at least six months with event_time and handle late events using allowed lateness plus a correction job. Add a parity test in CI that generates a synthetic event stream, computes features in both paths, and fails if distributions or values diverge beyond a tolerance. At serving, log the exact feature vector used for every decision for a sample of traffic. Validate with a temporal holdout, not a random split, and compare offline AUC against the replayed online AUC. Only promote if the parity test passes and the gap is within an agreed tolerance, for example two AUC points. Roll out as a shadow model before replacing the champion.