Custom Channel Integration Fails Under Load
A telecom integrates Adobe Campaign with an in-house SMS gateway via a custom external account and a JSSP page that Campaign calls from a delivery workflow. The integration worked in UAT at 500 messages/minute. In production, at 8,000 messages/minute, roughly 12% of sends return a gateway timeout, Campaign retries them, and the gateway reports duplicate message IDs. The JSSP page opens a new HTTP connection per call and has a 30-second timeout. The gateway's own documented limit is 6,000 requests/minute and it supports idempotency keys. Campaign's delivery logs show retries clustered in 90-second bursts every few minutes, suggesting the workflow's throughput is bursty rather than smooth.
1. Separate the failure into its contributing causes and rank them by how much of the 12% they explain.
Read the worked answer
Three causes compound here, and they interact:
- Throughput exceeds the gateway's documented ceiling. 8,000/min against a 6,000/min limit means at least ~25% of requests must fail or queue regardless of anything else. This is the dominant cause and no amount of retry tuning fixes it.
- Bursty workflow throughput. The 90-second retry clusters show Campaign is not sending at a steady rate; it pushes hard, hits timeouts, retries in a wave, and repeats. Bursts amplify cause 1 by concentrating load above the average.
- No idempotency. The gateway supports idempotency keys but the JSSP page does not send them, so every retry of a request that actually succeeded at the gateway creates a duplicate. This explains the duplicate message IDs - it is a consequence of retries, not an independent fault.
Ranking: the ceiling breach explains the failures; the burst pattern explains their clustering; the missing idempotency key explains the duplicates. Fixing only the timeout (a common first instinct) would reduce failures slightly while leaving duplicates and the ceiling problem intact.
2. Design the corrected integration and state how you would validate it before go-live.
Read the worked answer
Corrected design:
- Throttle below the ceiling. Cap the workflow's throughput to a safe margin under 6,000/min - e.g., 4,500/min - using a controlled send rate rather than relying on retries to self-regulate.
- Smooth the rate. Replace the bursty pattern with a steady pacing so the gateway sees a flat load; this removes the 90-second retry waves.
- Send idempotency keys. Generate a deterministic key per message (delivery ID + recipient ID + attempt-independent token) so a retried request is recognized as the same message. This makes retries safe by construction.
- Reuse connections. Replace the per-call HTTP connection with a pooled/reused connection and a timeout tuned to the gateway's real p99 latency, not an arbitrary 30s.
- Bound retries. Use a retry policy with backoff and a cap, and route exhausted messages to a quarantine workflow for inspection rather than retrying indefinitely.
Validation:
- Load test at 1.5x the target rate (≈6,750/min attempted, throttled to 4,500) against a gateway stub that enforces the 6,000/min limit and rejects duplicates.
- Assert: zero duplicate message IDs at the stub, failure rate below an agreed threshold, and steady request rate with no 90-second clustering.
- Run a chaos case - stub returns timeouts for 60s mid-run - and confirm retries produce no duplicates and the quarantine path captures only genuinely failed messages.
- Compare delivery log throughput curve before and after to prove the burst pattern is gone.