Feature Store Backfill for a Fraud Model
A fraud team trains a gradient-boosted model on 14 months of transaction data stored in Amazon S3. Offline features are computed nightly with a SageMaker Processing job and written to a feature group in SageMaker Feature Store. The online store serves the same feature group to a real-time endpoint. During a shadow test, the endpoint's predictions diverge sharply from batch scoring on the same transactions. Investigation shows that the nightly job recomputes a 30-day transaction-count feature using event_time set to job start time rather than the transaction timestamp. Also, three upstream fraud-analyst label corrections were applied only to the offline table. The team has 60 days of audit data and must retain point-in-time correctness for retraining.
1. Diagnose the two sources of training-serving skew and specify the concrete Feature Store configuration changes required to fix each.
Read the worked answer
Skew source 1 - timestamp semantics. The 30-day count is an aggregate whose value depends on when it is evaluated. By stamping event_time with job start, every record for a given run receives an identical timestamp, so point-in-time joins during training retrieve aggregates that would never have existed at inference time. The feature's value is therefore correct for the job, wrong for the transaction.
Fix: set event_time to the transaction's own timestamp and compute the aggregate as of that instant. In Feature Store, event_time is the key used by get_record/offline point-in-time joins; it must represent when the observation occurred, not when it was materialized. Also add a write_time audit trail so corrections are traceable without overwriting history.
Skew source 2 - label corrections applied only offline. The online store still holds stale feature values, and the offline table was mutated in place, destroying the ability to reconstruct what the model saw at training time.
Fix: treat corrections as new records with a later event_time (append-only). Re-ingest the corrected values into both offline and online stores from the same pipeline, and use PutRecord with the corrected event timestamp so the online store's latest-value lookup resolves to the corrected record while offline history remains intact for point-in-time reconstruction.
Validation: run a skew check comparing online get_record output against offline point-in-time retrieval for a sample of recent transactions before promoting the endpoint.
2. The team wants to guarantee that retraining 60 days from now reproduces exactly the feature values used today. What retention and lineage controls should they put in place?
Read the worked answer
Retention. The offline store must retain all record versions for at least the audit window plus the longest training lookback. Because corrections are appended rather than overwritten, the original and corrected values coexist; deleting older versions would make historical reconstruction impossible. Configure the offline store's S3 data catalog with a retention policy covering the full 60-day audit period and any retraining window beyond it.
Lineage. Record, per training run, the Feature Store offline table version or a snapshot reference, the query used for point-in-time joins, and the transformation code commit hash. Persist these as SageMaker experiment tracking artifacts or lineage entities so a future run can be tied to the exact inputs.
Determinism. Pin the aggregation window definition and time zone; floating-point aggregates over different partition boundaries can differ slightly. Materialize the training dataset as an immutable snapshot in S3 rather than re-querying the live offline store at retrain time - this converts a moving target into a fixed artifact and makes reproduction trivial.
Monitoring. Emit data-quality baselines per feature and alert on distribution drift between the snapshot and current offline data, so silent upstream changes surface before they contaminate a retrain.