The Nightly Batch That Cannot Meet the Fraud Window
A payments processor detects fraud with a nightly batch that scores the prior day's transactions. Chargebacks are rising and the risk team says decisions must be made within 2 seconds of authorization. The current pipeline: transactions land in an operational database, are extracted at 01:00, transformed through 40 SQL jobs, and scored by a model refreshed weekly. Peak volume is 900 transactions/second; the batch window is already 5.5 hours against a 6-hour SLA. The team proposes replacing the batch with a streaming platform and event-driven jobs. Two of the 40 jobs contain undocumented business rules that a retiring analyst wrote; nobody can explain them. The fraud model vendor requires feature inputs computed over a 30-day window.
1. Design the target integration architecture and identify the critical risks in getting there.
Read the worked answer
Target architecture. The 2-second requirement forces inline scoring at authorization, which the batch cannot provide at any tuning. Recommended:
- Event backbone: transactions published as events at authorization; a stream processor computes real-time features and calls the scoring service synchronously within the 2-second budget.
- Dual-path features: the 30-day windowed features cannot be recomputed per event in 2 seconds at 900 TPS. Maintain them in a low-latency store (e.g., a feature store) updated incrementally by the stream processor from the event log, with the batch retained only to recompute/backfill and to reconcile the incremental state.
- Decision and audit: every score, its features, and the decision must be persisted for dispute and model-governance evidence - a regulatory requirement in payments.
- Batch retained for non-latency work: settlement, reporting, and model retraining stay batch; streaming replaces only the decision path.
Critical risks:
- The two undocumented jobs. They may encode fraud-relevant rules (e.g., velocity thresholds, merchant exclusions). Migrating without understanding them risks silently removing controls. Mitigation: reverse-engineer via test-data diffing against the legacy output, and treat divergence as a blocking defect. Do not proceed on 'it looks fine.'
- Feature consistency. Incremental and batch-computed features will drift; without reconciliation, the model scores on subtly different inputs than it was trained on. Build a daily batch-vs-stream feature parity check.
- 2-second budget under peak. Measure end-to-end p99, not average; a 900 TPS peak with a slow feature lookup will breach. Load-test at 1.5x peak.
- Dual-run period. Run batch and streaming in parallel, comparing decisions, before the batch is retired.
2. The vendor's 30-day feature window conflicts with the 2-second latency target. How would you resolve this, and what would you tell the risk committee?
Read the worked answer
The conflict is real but resolvable by separating computation from lookup. The 30-day window does not need to be computed at decision time; it needs to be available at decision time. The stream processor maintains a rolling 30-day aggregate incrementally as each event arrives, so the scoring call is a key lookup, not a 30-day scan. The batch job's role shifts to nightly reconciliation and backfill of the incremental state - catching missed or out-of-order events.
What to tell the risk committee, plainly:
- Fraud decisions move from next-day to sub-2-second, which is the change that reduces chargebacks.
- The model's inputs are unchanged in definition; only the computation method changes. We will evidence parity with a daily comparison between stream-maintained and batch-recomputed features, and we will not cut over until parity holds across a defined tolerance for a sustained period.
- There is a residual risk: out-of-order or late-arriving events can make the incremental window briefly stale. We will define a maximum acceptable staleness and alert when exceeded, and the batch remains the system of record for dispute reconstruction.
Honest caveat. If the vendor's model requires exact recomputation semantics that incremental aggregation cannot guarantee (e.g., order-dependent logic), the correct answer is to raise it with the vendor and possibly accept a short window for the real-time path while the full window is used for post-authorization review. Do not promise exact equivalence without evidence - the committee needs the residual risk stated, not hidden.