Cold Starts on a Scale-to-Zero Fraud Endpoint
A fraud-scoring scikit-learn model is deployed to a Databricks Model Serving endpoint. Peak traffic is 20 requests/second; the business requires p95 < 200 ms. The endpoint uses workload size Small, min replicas 0, max replicas 6, and scale-to-zero. The model's load_context reads a 1.2 GB customer-history Parquet table from a Unity Catalog volume. After 10 minutes idle, p95 rises to 1.2 s for several minutes and the first requests time out. Under steady load, p95 is 180 ms and CPU is 55%. The team wants to keep costs low.
1. What is causing the latency spikes, and which endpoint/model changes would you make? Discuss cost trade-offs.
Read the worked answer
The spike is a cold-start problem with two components: scale-to-zero means no warm process exists, and each new replica must load a 1.2 GB Parquet table inside load_context before serving. Steady-state p95 of 180 ms against a 200 ms SLO also leaves little headroom, so even a modest CPU increase during warm-up breaches the target.
The first change is to set min replicas to at least 1, which removes the idle-to-first-request cold start; min replicas 2 gives availability during restarts but doubles idle cost. The second is to shrink the startup payload: reduce the lookup to only required columns and rows, store it in a compact format, or precompute the needed customer features into the model artifact or online feature store. If CPU at 55% is a true steady-state figure, Small may be sufficient after startup is optimized; Medium adds headroom but may be over-provisioned. The cost trade-off is explicit: paying for one always-warm replica is usually cheaper than violating a fraud SLO.
2. How would you safely roll out v2 with a 5% canary and evaluate it?
Read the worked answer
Use one endpoint with two served entities rather than a second endpoint. Serve v1 at 95% and v2 at 5% traffic, with inference tables enabled for both. Keep model aliases such as champion and challenger so traffic configuration references aliases, not raw version numbers.
Evaluate the canary on three axes: latency and error rate from the endpoint, prediction distribution and feature null rates from inference tables, and business outcomes once labels arrive. A 5% canary at 20 requests/second gives roughly one request per second, which may be enough for latency but too small for immediate label-based AUC; plan to run it long enough to accumulate outcomes. If v2 has worse p95, higher error rate, or a shifted score distribution, move 100% back to v1 by changing traffic weights or the alias. If it passes, increase traffic in steps rather than jumping directly to 100%.