Real-time scoring at 2,000 requests per second
A telecom wants to score 2,000 requests per second at peak with p99 latency below 50 ms using a gradient boosting model in Watson Machine Learning on IBM Cloud Pak for Data 4.8. The current CP4D environment has dev (3 workers at 16 vCPU), test (3 workers), and prod (6 workers at 32 vCPU/128 GB). The model uses 120 features served from a Db2 feature store and Redis. The data science team wants to deploy it as a Python function. The current WML online deployment gives 180 ms p99 at 400 requests per second with 3 replicas. The network team can add a global load balancer. The business requires canary releases and drift monitoring, and the model must be retrained weekly.
1. Diagnose the latency and throughput gap and propose a deployment architecture.
Read the worked answer
The gap is roughly 5x throughput and 3.6x lower p99. Three replicas at 400 requests per second means about 133 requests per second per replica; reaching 2,000 requires at least 15 replicas at the same efficiency, and that is before accounting for the Python function overhead and per-request feature lookups. A Python function deployment is a poor fit for this latency budget because interpreter startup, package loading, and synchronous Db2 calls add tail latency. Move to a supported framework runtime for the model, pre-load the model and encoders in memory, and fetch features from Redis or an in-process cache rather than Db2 on the request path. Batch feature retrieval or precompute the feature vector where possible. Scale the WML deployment across multiple prod workers with anti-affinity, set CPU requests and limits, and load-test at 2,000 requests per second with realistic feature distributions. If p99 remains above 50 ms, separate feature serving into its own low-latency service and keep WML focused on inference.
2. How do you implement canary releases and drift monitoring without breaking the 50 ms budget?
Read the worked answer
Use two WML deployments, champion and challenger, behind the global load balancer or WML traffic splitting if supported. Start the canary at a small percentage, compare business metrics and latency, and increase only after acceptance. For drift monitoring, do not make a synchronous call to a monitoring service in the request path. Log a sample of requests and predictions asynchronously to storage, then run model monitoring on the sampled data. Keep the sample rate low enough to protect latency and high enough to detect drift. Define drift thresholds and a rollback trigger before the canary starts. If the monitoring tooling requires inline scoring, reject that design; it will consume the latency budget. Validate that canary traffic does not overload Redis or the feature store.
3. What governance and rollback controls do you require for weekly retraining?
Read the worked answer
Weekly retraining should be automated but gated. Require a reproducible pipeline that snapshots training data, validates schema and feature distributions, trains the candidate, and evaluates it against offline and holdout metrics. Register the model with version, data lineage, and evaluation results, and require approval before promotion. Compare the candidate with the current champion on a shadow or canary deployment, not directly on all traffic. Define rollback to the previous model version as a one-step operation with a tested runbook. Monitor post-deployment accuracy, drift, and latency; if any threshold is breached, revert. Also confirm that retraining does not change feature definitions without coordinating the online feature store, because a model that expects new features will fail at scoring time.