Catastrophe peak: removing synchronous fraud scoring
A national auto insurer processes 8 million claims a year on Pega. Normal volume is 22,000 claims per day, but catastrophe events drive a 4x peak for 48-72 hours. The system supports 3,000 concurrent adjusters. At month-end, the primary database reaches 85% CPU, reporting queries time out after 90 seconds, and a queue processor for claim document generation backs up to 120,000 items. The external fraud-scoring REST service has a p95 latency of 2.5 seconds and occasional 30-second timeouts; claim submission currently calls it synchronously before creating the case. Adjusters complain that the claim list takes 8 seconds to load. The CIO wants to remove the synchronous dependency and ensure no claim is lost during peaks. There is a 4-hour recovery point objective and a 30-minute recovery time objective.
1. Diagnose the likely bottlenecks and propose a target processing architecture for claim intake and document generation.
Read the worked answer
The synchronous fraud call is the first bottleneck: a 2.5-second p95 call, sometimes 30 seconds, holds the case-creation transaction and a database connection, so peak arrivals amplify connection-pool pressure. The 8-second claim list and 90-second report timeouts point to unindexed or overly broad list queries and reporting on the primary database. The 120,000-item queue backlog suggests a single queue processor or a slow downstream document service. I would persist the claim immediately, return a case ID, and move fraud scoring to an asynchronous queue processor with correlation ID, retry, and dead-letter handling. Document generation should also be asynchronous, partitioned by region or claim type so one blocked partition does not stop all claims. Reference data should use data pages with caching, not repeated database reads. Reporting should move to a read replica or reporting database, and adjuster lists should use optimized report definitions with pagination and indexed filters. The tradeoff is eventual consistency: the claim exists before fraud completes, so the case status must show pending fraud check.
2. How do you guarantee no claim is lost and no duplicate fraud charge is created when the fraud service times out?
Read the worked answer
Use an idempotency key derived from the claim ID and submission request ID. The queue processor writes a fraud request record before calling the service, with status Pending, and updates it to InProgress, Completed, or Failed. If the call times out, do not assume failure; retry with the same idempotency key and exponential backoff. The fraud service must honor that key, or Pega must check a response record before retrying. After the retry limit, route to a work queue for manual reconciliation and keep the claim in a visible exception status. Never delete the request record. For catastrophe peaks, cap concurrency per downstream service and use a circuit breaker so the queue absorbs load rather than the database. This meets the 4-hour RPO because the request record is persisted before the external call; the tradeoff is more state and reconciliation logic, but it prevents lost claims and duplicate charges.
3. What scaling and monitoring design would you put in place for a 4x, 72-hour catastrophe peak?
Read the worked answer
Scale the web and application tier horizontally, but treat the database and queue processors as the constrained resources. Partition queue processors by region or claim type and run multiple instances, with back-pressure when queue depth exceeds thresholds. Use a separate reporting database so month-end queries cannot consume primary CPU. Pre-scale before predicted events and load-test at 4x with realistic 2.5-second downstream latency. Monitor case creation rate, queue depth, oldest message age, external p95 and p99, error rate, database CPU, connection pool saturation, and adjuster page response time. Alert on queue age, not just queue count, because 120,000 items may be acceptable if the oldest is 2 minutes but not if it is 2 hours. The 30-minute RTO requires active-passive or multi-zone deployment and tested failover; the 4-hour RPO requires persisted queue records. I would accept eventual document generation, but not eventual claim persistence.