High-volume consent sync against a rate-limited vendor API
A marketing operations team must synchronize 250,000 consent records to a vendor’s REST API every night. The vendor enforces 50 requests per second per tenant and a burst of 100; excess calls receive HTTP 429 with a Retry-After header. About 5% of calls also return transient 503 errors. The synchronization window is 2 hours, after which the vendor’s nightly import closes. Each record is a small JSON payload; the API supports an upsert keyed by the vendor’s external contact ID, but not bulk upload. The current CAI process processes records sequentially and takes about 20 hours; when it fails, operators restart from the beginning, causing duplicate consent events and occasional consent revocations to be re-applied. The business cannot accept lost revocations or duplicate audit entries. Infrastructure can provide up to 16 concurrent workers but the vendor relationship manager warns that sustained traffic above 45 requests per second may trigger a tenant review.
1. Calculate the throughput needed and propose a concurrency and rate-limiting design that finishes within the window with headroom. Show the arithmetic.
Read the worked answer
250,000 records in a 2-hour window (7,200 seconds) requires 250,000 / 7,200 = 34.7 requests per second average if one record is sent per request. At 45 requests per second, the floor is 250,000 / 45 = 5,556 seconds, or about 92.6 minutes, leaving roughly 27 minutes of headroom for retries and checkpointing. At the vendor’s 50/s cap, the floor is 5,000 seconds (83.3 minutes), but that leaves little room for 429s.
Target 40 requests per second with a token-bucket or rate limiter shared by all workers, not a per-worker limit. With 8-12 concurrent workers and a 40/s aggregate cap, each worker averages 3-5 requests per second, which is realistic for small JSON calls. Do not set 16 workers each at 10/s; that would attempt 160/s and trigger tenant review. Monitor achieved rate and 429 rate; reduce to 35/s if 429s persist. If the API later supports bulk, re-baseline; it does not today.
2. How should the process handle 429 and 503 responses, and why is blind retry dangerous for consent revocations?
Read the worked answer
429 means the vendor’s rate limit was exceeded; honor Retry-After and pause the shared limiter, not just the individual worker, otherwise other workers keep hitting the limit. 503 is transient; retry with exponential backoff and jitter, capped at a small number of attempts, for example 5, and a total deadline.
Blind retry is dangerous for consent revocations because the operation is not naturally idempotent in business terms: replaying an old opt-out after a newer opt-in can revoke consent that was legitimately restored. Protect every write with the vendor external contact ID plus a consent version or timestamp; the upsert must reject stale versions. Persist the version being sent and compare it on retry. If the vendor API cannot enforce version ordering, do not retry a revocation automatically after an ambiguous timeout; route it to a reconciliation queue for a human or a read-before-write check. Retries must not create duplicate audit entries; include a request ID the vendor can deduplicate.
3. Design checkpointing and restart so a failure at record 180,000 does not restart from zero or duplicate audit events.
Read the worked answer
Process records in chunks, for example 500, and commit a checkpoint only after the chunk is acknowledged by the vendor. The checkpoint should store the last successfully processed contact ID or a stable watermark plus the consent version, not just a row number, because row order can change. On restart, query the checkpoint and resume from the next unprocessed record.
Because the vendor upsert is keyed by external ID, replaying the last incomplete chunk is safe only if the payload carries the same version; combined with the version check, this gives effectively-once business behavior. Write an audit record per contact with correlation ID, version, HTTP outcome, and retry count; make audit writes idempotent on (contact_id, version, operation). Do not keep the checkpoint only in process memory or a single process object that can be overwritten by a concurrent run. Use a run lock so two nightly jobs cannot process the same range.