Latency Spikes When Batch and Real-Time Share the Cluster
A bank runs Pega CDH for 12 million retail customers. Inbound web and mobile containers require a p95 response under 150 ms. A nightly outbound batch scores all 12 million customers and writes 40 million offers to a campaign database. The production cluster has three CDH nodes and one Decision Data Store node. During the 22:00-02:00 batch window, inbound p95 latency rises to 1.9 seconds and container timeouts reach 4%. The batch job and real-time containers share the same Decision Data Store and the same data flows. Member profile updates arrive continuously from a Kafka topic and are applied by a real-time data flow. The bank wants to add a second daily batch at 06:00 and a new credit-limit-increase proposition that uses a predictive model with 45 predictors.
1. Identify the architectural bottlenecks and propose a target architecture that separates batch and real-time workloads.
Read the worked answer
The single Decision Data Store node is the main bottleneck. Batch scoring of 12 million customers and writing 40 million offers competes for the same CPU, I/O, and connections as inbound reads. Shared data flows also compete for resources. The 06:00 batch would overlap with morning peak and repeat the problem.
Separate batch and real-time workloads: use separate node roles or a separate batch cluster, and give the Decision Data Store multiple nodes with replicas. Precompute batch scores into a read-only decision data set rather than recalculating at decision time. Stream Kafka updates through a real-time data flow with backpressure, and keep reference data cached. Schedule batch outside peak windows or isolate it. Add container timeouts and circuit breakers so slow dependencies do not exhaust threads. Size the Decision Data Store for read and write throughput and monitor latency.
2. How should the new proposition's 45 predictors be managed to protect the 150 ms SLO?
Read the worked answer
Do not compute 45 predictors at decision time from external systems. Precompute them in data flows into the Decision Data Store or a decision data set, and at decision time look them up by customer ID. Cache model outputs where appropriate. If the predictive model is too heavy, reduce the feature set or use a precomputed score as a single predictor.
For truly real-time events, use a stream with last-known-value semantics and a short timeout. Run load tests with production-like volumes and measure p95. Keep the model as a separate component so it can be monitored and swapped. If the SLO still cannot be met, move the credit-limit-increase proposition to batch or asynchronous channels.
3. What monitoring and fallback would you put in place for the Kafka profile updates?
Read the worked answer
Monitor consumer lag, data flow error rates, and Decision Data Store write latency. Route malformed messages to a dead-letter queue and alert on lag above a threshold. Reconcile counts between Kafka and the Decision Data Store.
Fallback to the last known profile in the Decision Data Store when a real-time update is delayed or fails. Use a circuit breaker so the container can return a default or lower-priority offer rather than timing out. Include an operational dashboard and runbook for lag spikes, and test failover before the 06:00 batch is added.