Real-Time Fraud Scoring Under a 120 ms Latency Budget
A retail bank has a fraud scoring model used at online checkout. The current model is a 2.1 GB ensemble of 40 gradient-boosted trees plus a feature pipeline that joins 12 real-time aggregates from Redis and a customer profile table. The bank wants to move from nightly batch scoring to real-time scoring under a 120 ms p99 latency budget at 900 requests per second during peak. A prototype on a single 8-vCPU VM achieves p99 310 ms, with 70% of time spent in feature joins and 20% in model inference. The platform team proposes Kubernetes with 20 replicas and a 5-minute rolling deployment. Risk wants a canary release and the ability to fall back to the previous model within one minute. Compliance requires the exact model version and feature values to be logged for every declined transaction.
1. Where would you focus optimization to meet the 120 ms p99 budget, and what architecture would you recommend?
Read the worked answer
Profile first, then remove the biggest cost. At 70% feature joins, the model is not the main problem. Precompute the 12 real-time aggregates in a feature service and fetch them with a single multi-get; avoid per-feature network calls. Cache customer profile data with a short TTL and use local in-process caches for hot keys. Reduce the join fan-out by denormalizing stable attributes into a single wide feature row. For inference, consider compiling the tree ensemble, reducing tree count via distillation, or using a lighter model for the first-pass decision with a secondary model only for borderline cases. A 2.1 GB ensemble at 900 requests per second is expensive; quantize or prune if accuracy holds. Split batch and real-time paths: log asynchronously to a queue so compliance logging does not sit in the latency path. With those changes, 120 ms p99 is plausible on fewer than 20 replicas.
2. Design the deployment and rollback approach that satisfies risk and compliance.
Read the worked answer
Use a canary with shadow mode first: send a copy of live traffic to the new model, compare scores and latency without affecting decisions. Then route 1%, 5%, 25%, and 100% with automated gates on latency, error rate, score distribution, and decline rate. Put the model version and feature-schema version behind a feature flag so rollback is a configuration change, not a redeploy. Keep the previous model warm in the same cluster. Risk's one-minute fallback means the flag must switch traffic at the load balancer or service mesh, and the old model must remain provisioned.
Compliance logging should be asynchronous but durable: write an immutable event with model version, feature values, score, and decision to a replicated log before responding, or use a write-ahead queue with guaranteed delivery. Test rollback in game day exercises.
3. What monitoring and handoff artifacts are needed for production operation?
Read the worked answer
Define SLOs for p99 latency, error rate, and availability, with alerts at 80% of budget. Monitor score distribution, decline rate, feature null rates, and drift for each feature. Track data freshness for Redis aggregates and profile tables. Keep a model card with training data, intended use, performance by segment, and known failure modes. Provide a runbook for latency spikes, model errors, and rollback. Version the feature pipeline and model together; store reproducible feature snapshots for audits. Set up a weekly review of declined transactions and false positives, and a monthly review of drift and retraining. Handoff should include an on-call rotation, escalation path, and a named owner for the feature service. Compliance requires retention and access controls for the logged feature values.