Tool-layer backpressure: 500 agents against a 100 requests/second CRM API
Vertex Ops runs up to 500 concurrent agent sessions that call a CRM REST API to read and update customer records. The contract limits the tenant to 100 requests per second overall and 20 per second on write endpoints. Agents retry failures three times with a fixed 200 ms delay and no jitter. During a marketing campaign, agents fanned out to 40 parallel sub-tasks each; the API returned 429s, retries roughly tripled the offered traffic, and p99 task latency went from 14 s to 6 minutes. Afterwards, 3% of records carried duplicated notes. The framework's default is one sequential tool call per step, so a task needing 12 record lookups takes 12 round trips. A Redis cluster is available, and the CRM exposes a batch read endpoint accepting up to 50 IDs. Product wants the campaign flow to survive the next peak without a contract renegotiation.
1. Design the rate-governance layer. Where must the limit be enforced, and how should retries behave?
Read the worked answer
Enforcement belongs in a gateway in front of the CRM, not in prompt instructions to the agents. A per-tenant token bucket held in Redis (100/s read, 20/s write, separate buckets) is shared by all 500 sessions, so the aggregate is capped regardless of how many agents fan out. Each agent also gets a per-task tool-call budget, which bounds the 40-way fan-out at its source.
Retries: the current fixed 200 ms delay synchronises all clients into repeated waves. Replace with exponential backoff and full jitter (base 500 ms, cap 8 s, maximum 3 attempts) and honour Retry-After. Cap the retry budget at roughly 10% of calls; when the budget is exhausted, fail the step explicitly rather than continuing to amplify load. Add a per-endpoint circuit breaker that opens on sustained 429/5xx and routes the agent to a degraded path - park the task and resume on the next turn - instead of blocking a worker.
For writes, require idempotency keys derived from task id, record id and operation, so a retried write is deduplicated server-side; that is the only reliable fix for the duplicate notes, since client-side retry logic alone cannot make a non-idempotent POST safe.
Verify with a load test that replays the campaign fan-out pattern and asserts offered traffic never exceeds the bucket limits.
2. Cut the round trips without exceeding the rate limits. What changes in the agent's tool design?
Read the worked answer
Twelve sequential lookups is a tool-surface problem, not a model problem. Three changes:
- Batch reads. Expose
fetch_records(ids[])backed by the CRM's 50-ID endpoint. The planner first resolves which records it needs, then issues one or two batched calls instead of twelve. Rate accounting should charge a batch as its constituent record count, otherwise batching silently bypasses the 100/s cap. - Bounded parallel fan-out. Where sub-tasks are genuinely independent, execute tools concurrently through a semaphore sized against the token bucket (for example 8 in flight), not 40. Independence must be established before parallelising: two writes to the same record must be serialised, which is how duplicate notes appear.
- Read-through caching in Redis with a short TTL keyed on record version. Repeated lookups within a task and across sessions in the same campaign collapse to one call; writes invalidate the key.
Also expose a single append_note(record_id, text, idempotency_key) tool rather than a generic update, so the agent cannot read-modify-write a note list and resurrect stale content. Measure tool calls per task before and after - the target is a low single-digit number for a 12-record task.
3. How would you detect and repair the duplicated notes, and what would prevent recurrence?
Read the worked answer
Detection: query notes created in the campaign window, hash normalised note text per record, and flag records with more than one identical or near-identical note within a short time band. Cross-check against the idempotency key if one was logged, and against the audit trail of tool calls, so duplicates that differ only in whitespace or a timestamp suffix are still caught. Produce a per-record report rather than a bulk delete, because not all duplicates are identical - some are legitimate follow-ups.
Repair: soft-delete the redundant copies, keep one canonical note, and notify record owners. Do this behind a reviewed, reversible job, never an automated destructive sweep, and record what was removed and why.
Prevention: idempotency keys on every write, serialised writes per record id, and a post-condition check that reads back the record and asserts the expected note count before the task is marked complete. Add a campaign-scale replay to the regression suite that reproduces the fan-out pattern with the new gateway and asserts zero duplicates plus offered traffic within the 100/s and 20/s buckets.
Finally, make retry and fan-out settings configuration-reviewed rather than framework defaults - the failure originated in defaults nobody chose.