Choosing Between a Synchronous Call and an Event-Driven Handoff
A subscription business has a Boomi process that, when a customer cancels, must (a) update the billing system, (b) notify the CRM, (c) post to a data warehouse, and (d) send a confirmation email. Today all four are synchronous calls inside one process, executed in sequence. The billing call averages 300 ms but has p99 of 9 seconds during month-end. The email provider is frequently slow and has twice caused the entire cancellation to fail and roll back, leaving the customer still subscribed. Cancellations arrive at about 200 per hour normally and 2,000 per hour at month-end. The business requires that a cancellation be reflected in billing within 5 seconds of the customer's request.
1. Restructure this process. Which steps must remain synchronous, which should be decoupled, and how does the 5-second billing requirement constrain the design?
Read the worked answer
Only billing must be synchronous, because it is the sole step with a hard 5-second commitment and it is the step whose failure has real customer consequence - a customer who cancelled and is still being billed. CRM notification, warehouse posting, and email are all consequences of a completed cancellation, not preconditions for it, and none has a tight latency requirement. They should be decoupled from the cancellation path.
The design: the inbound request validates and calls billing synchronously, returns success once billing confirms, and publishes a cancellation event carrying the billing confirmation. Separate consumers handle CRM, warehouse, and email, each with its own retry and dead-letter handling. This removes the email provider's latency from the critical path entirely - the failure that left customers subscribed becomes structurally impossible, because email failure can no longer roll back billing.
The 5-second requirement also forces a decision about billing's p99 of 9 seconds. At month-end, some calls will exceed 5 seconds. The process needs a bounded wait and a defined behavior on timeout - either asynchronous confirmation with a pending state the customer can see, or a fast retry - rather than blocking indefinitely. Measure billing p99 during month-end before committing to a synchronous-only design; if it cannot meet 5 seconds, the requirement must be renegotiated or billing needs its own capacity work.
2. Month-end load reaches 2,000 cancellations per hour. What breaks first in your design, and how do you test for it before month-end?
Read the worked answer
Billing capacity breaks first. 2,000 per hour is roughly 0.55 per second average, but month-end traffic is bursty and billing's p99 already stretches to 9 seconds. If cancellations cluster, concurrent billing calls rise sharply and the slow tail lengthens, pushing more requests past the 5-second bound. The synchronous path has no buffer, so excess load becomes timeouts rather than queueing.
The decoupled consumers break second, but less visibly: the event backlog grows. CRM and warehouse consumers falling behind is tolerable for a while, but a growing backlog means the dead-letter path gets exercised under pressure and failures compound.
Test with a load profile that reproduces the burst shape, not just the average - replay a month-end hour against a staging environment with realistic billing latency injection, including a 9-second tail. Watch concurrent billing calls, timeout rate, and consumer lag. Verify the dead-letter path actually captures and replays failures, since it is usually untested until it is needed. Confirm billing has headroom or a queue in front of it; if it does not, the honest answer is that the 5-second commitment cannot be met at month-end without capacity changes, and that should be surfaced before the next cycle rather than discovered during it.