I85PREP
⌘ K
AWS Machine Learning Specialty study materials
AWS Machine Learning Specialty study resources

AWS Machine Learning Specialty (AWS): Fraud Scoring: When the Online and Offline Feature Paths Diverge

AWS Machine Learning Specialty · Put your knowledge to work in context.

All case studies

Fraud Scoring: When the Online and Offline Feature Paths Diverge

A mobile payments company scores card transactions in real time. The endpoint must return a fraud score in under 100 ms. Peak traffic is 40,000 events/s (average 8,000), each event about 1.2 KB. Key features are a 5-minute transaction count and a 30-minute spend sum per card. Roughly 4% of events arrive late, by up to 15 minutes, because the mobile SDK buffers when offline. A nightly AWS Glue job recomputes every feature from the raw S3 archive and writes the training table. The online path computes the same features separately, in a Lambda function backed by a DynamoDB counter. Retrained models consistently underperform in production, and offline AUC has fallen from 0.88 to 0.71 over three months even though neither the model code nor the feature list has changed.

1. The team concludes that user behaviour has drifted. Evaluate that conclusion and describe the pipeline you would build instead.

Read the worked answer

The conclusion is probably wrong. Offline AUC fell with unchanged code, so look for a change in the relationship between the two feature implementations rather than in the users.

  • The nightly Glue job sees the complete event set for a window, including the 4% that arrive up to 15 minutes late, and places them in the window they belong to by event time.
  • The Lambda/DynamoDB path increments a counter when the event arrives. Late events land in the wrong window or are dropped, and DynamoDB counters have no event-time ordering.
  • Labels are produced from the batch table. So offline AUC is measured on features that are strictly more informative than what is served online. As the offline-SDK population grew, the late-event rate and the skew grew with it, which also shows up as a distribution shift on the feature.

Target architecture - one definition, two stores:

  1. Compute the 5-minute and 30-minute aggregates in Kinesis Data Analytics for Apache Flink, using event-time windows, watermarks and allowedLateness of 15-20 minutes. One implementation only.
  2. Write results to SageMaker Feature Store: the online store serves GetRecord for the endpoint; the offline store accumulates the history used for training.
  3. Build training sets with a point-in-time join (create_training_set on entity ID + label event_time) so each row only sees feature records with event_time <= the label timestamp.
  4. Verify with a replay harness: push a day of archived Kinesis records through the streaming job, write to the offline store, and diff against what the batch job produces. Any residual difference is the skew you have not fixed.

2. Size the Kinesis data stream and identify the partitioning pitfall for this workload.

Read the worked answer

Throughput math: 40,000 rec/s x 1.2 KB = 48 MB/s at peak.

Per-shard write limits are 1 MB/s or 1,000 records/s, whichever binds first:

  • Records: 40,000 / 1,000 = 40 shards
  • Bytes: 48 / 1 = 48 shards

Bytes bind, so the floor is 48 shards. With headroom for skew and spikes, start at 72-80 provisioned shards, or use on-demand mode and verify against the documented default throughput ceiling for your account.

The pitfall: keying by card_id alone creates hot shards. A corporate card can produce thousands of transactions per minute; the 1,000 rec/s limit is per shard and is not shared across the stream, so one hot key throttles the whole application while other shards sit idle. Naive salting (card_id + random bucket) fixes the hotspot but destroys per-card ordering. Since the aggregates must be per card, keep enough key cardinality and do the ordering inside Flink with keyed state on card_id (which handles cross-shard arrival via event time anyway).

Also plan the consumers: if you need both the Flink aggregation and an archival Firehose/S3 sink, use enhanced fan-out so each consumer gets dedicated read throughput instead of sharing the 2 MB/s per-shard read budget.

3. How do you keep the offline store point-in-time correct when the streaming job emits late-but-corrected window values?

Read the worked answer

Late data means a window that was already emitted can change. Two rules make that safe:

  1. Emit corrections with the original event time, never with processing time. A corrected 5-minute count for window [10:00, 10:05) that is computed at 10:21 must still be written with the window's event timestamp. The Feature Store offline store is append-only, so you get a second record for the same (card, event_time). The point-in-time join takes the latest ingested record whose event_time <= the label timestamp, so training rows pick up the correction automatically and remain correct.
  2. Set the label cutoff after the allowed lateness. If a label is timestamped at 10:05 and corrections for that window can still land 20 minutes later, training rows labelled near the boundary will sometimes see the pre-correction value. Either delay the label cutoff by the allowed lateness, or accept a small inconsistency that is bounded and measurable.

Verify rather than assume: replay a fixed day of traffic, run the job twice with different lateness settings, and confirm the training sets are identical apart from the expected corrections. Then confirm the online store only ever holds the latest value per entity, since inference has no notion of event time. Writing corrections with processing time is the specific mistake that reintroduces the leakage you just removed.