Streaming Feature Freshness vs. Replay Cost
A fraud-detection team runs a Kinesis Data Stream with 8 shards receiving ~4,000 records/sec at peak. A Lambda consumer enriches each record and writes to DynamoDB, which a SageMaker real-time endpoint reads as a feature store at inference. During a traffic spike, iterator age on two shards climbed to 90 seconds and the endpoint began scoring with stale velocity features, dropping recall on a holdout from 0.81 to 0.62. The team wants sub-5-second feature freshness for velocity counters while keeping a 30-day replayable history for retraining. Lambda concurrency is capped at 1,000 account-wide and other workloads use ~400. Records are ~2 KB and keyed by customer_id, and a small number of high-volume customers dominate traffic.
1. Diagnose why iterator age spiked and propose a sharding/partitioning change that improves freshness without unbounded Lambda concurrency growth.
Read the worked answer
The spike is a hot-key problem, not a raw throughput problem. 8 shards give 8 MB/sec and 8,000 records/sec of write capacity, and 4,000 records/sec at 2 KB is only ~8 MB/sec - right at the ceiling - but the symptom is concentrated on two shards. Because records are keyed by customer_id, a few high-volume customers land on the same shards and serialize; those shards hit their per-shard record limit while others idle. Lambda then backs off on the throttled shards, and iterator age grows.
Fix in two layers. First, salt the partition key so heavy customers spread across shards (e.g., customer_id + a hash bucket suffix), and raise shard count so aggregate capacity exceeds peak with headroom - target ~2x the observed peak record rate. Second, decouple consumption from the 1,000-concurrency Lambda cap: run the enrichment on a Kinesis Client Library consumer on ECS/EC2 with one worker per shard, or use enhanced fan-out so each consumer gets dedicated read throughput. Keep a separate Firehose delivery to S3 for the 30-day replayable history so retraining does not depend on the real-time path. Reserve Lambda for low-volume side tasks so the 600 free concurrency units are not the bottleneck.
The judgement point: adding shards alone does not fix hot keys, and raising Lambda concurrency alone does not fix per-shard serialization.
2. The team proposes computing velocity counters directly in the endpoint's request handler from DynamoDB on every call. Evaluate this against a precomputed feature approach.
Read the worked answer
On-demand computation in the request path couples inference latency to DynamoDB read latency and to the number of counters needed. At 4,000 records/sec peak with several counters per customer, the endpoint would issue many reads per request, raising p99 latency and read cost, and a DynamoDB throttle would surface directly as inference errors. It also makes the endpoint's behavior depend on a mutable table, so a bad backfill silently changes scores.
Precompute the counters in the stream consumer and write them as a versioned feature record with an event-time timestamp. The endpoint does one point read (or a cached read with a short TTL) and can reject or flag a feature older than the freshness SLA (e.g., >5s) rather than silently scoring stale. This gives an explicit staleness signal the team can alert on, which the on-demand approach hides.
Trade-off: precomputation adds pipeline lag and a second write path, and a consumer outage freezes features. Mitigate with a dead-letter path, a freshness alarm on iterator age, and a fallback that serves last-known-good features with a flag. If some counters are cheap and rarely change, computing those on demand is fine - the decision should be per-feature based on latency budget and update rate, not uniform.