Real-Time Fraud Endpoint: Capacity, Features, and Canary
A payments company deploys a fraud model as a real-time Databricks Model Serving endpoint. Peak traffic is 2,000 requests per second; overnight it falls to 150. The p99 latency objective is 50 ms including an 8 ms online feature-store lookup. Thirty features are fetched per request; most update hourly, but account velocity updates every five minutes. The model artifact is 2.5 GB; the endpoint currently uses a CPU medium workload with scale-to-zero and one instance. Cold starts take 25 seconds and cause timeouts after quiet periods. Each request payload is 12 KB, and inference tables log requests and responses. A new version 8 is ready for a 10% canary. The team is unsure whether to use GPU, how much warm capacity to keep, and how to decide promotion or rollback from canary evidence.
1. Estimate the required concurrency and recommend scaling and warm-capacity settings. Address GPU versus CPU and scale-to-zero.
Read the worked answer
Concurrency is approximately requests per second multiplied by latency. At 2,000 RPS and 50 ms, average in-flight requests are about 100. For p99 headroom, plan for 150 to 200 concurrent requests. If one medium instance safely handles about 10 concurrent requests at p99, that implies roughly 15 to 20 peak instances. At 150 RPS overnight, average concurrency is about 7.5, so one or two warm instances may suffice if they meet latency. Scale-to-zero is incompatible with a 50 ms p99 and a 25-second cold start; keep a minimum warm capacity. The 2.5 GB model fits CPU memory, so GPU is justified only if profiling shows GPU-bound inference. GPU increases cost and cold-start complexity. Load test with 12 KB payloads, the feature lookup included, and set autoscaling min and max from measured per-instance concurrency, not from average CPU alone.
2. The online feature lookup contributes 8 ms at p99. How would you protect the latency budget and handle feature freshness?
Read the worked answer
Treat the 8 ms as a hard budget line and measure it separately from model inference. Fetch only the 30 required features; do not add external API calls per request. Most features update hourly, so precompute and cache them; velocity features update every five minutes and must come from the online store or a low-latency cache. Validate feature order, defaults, and null handling in the endpoint. Log feature values to the inference table for drift analysis. If the online store slows down, fail fast or use a bounded-staleness fallback rather than silently substituting stale values without a flag. Co-locate the online store with the endpoint where possible. Test lookup p99 under peak concurrency, because an 8 ms single-request measurement can degrade under load.
3. Design the 10% canary and the promotion or rollback criteria. What evidence should come from the inference table?
Read the worked answer
Route 10% of traffic to version 8 and 90% to the current Champion, keeping both configurations versioned. Run the canary through at least one full traffic cycle and enough requests for stable estimates. Predefine promotion criteria: p99 latency at or below 50 ms, error rate no worse than the Champion, no increase in null or default feature rates, score distribution within an agreed PSI bound, and no degradation in fraud or approval rates on comparable slices. Use inference-table logs to compare request volume, response codes, scores, feature values, and latency by version. Roll back if p99 exceeds 60 ms, errors exceed the agreed threshold, or score distribution shifts materially. Rollback means shifting traffic back to the Champion alias or configuration, not deleting version 8. Keep the canary evidence and require a load test at 2,000 RPS before full promotion.