Custom Code Action Times Out at 4,000 Leads per Day

The design is synchronous per record, with no concurrency control, inside a 20-second execution window. Average volume is only about 2.9 contacts per minute, but the 9 a.m. and 1 p.m. bursts exceed 100 requests per minute and 10 concurrent requests. When the vendor responds slowly or returns 429, the code action hits the 20-second limit. Re-enrollment then runs the same contact again, which explains the 7% double enrichments.

Redesign around the vendor's asynchronous job queue:

1. The HubSpot custom code action should do one small thing: validate the contact, generate an idempotency key, and POST a job to the vendor or to an internal queue. It should not wait for enrichment results.
2. A worker outside HubSpot processes jobs at or below 100 per minute and 10 concurrent, respecting the vendor's limits.
3. The worker writes results back to HubSpot through the API, or the vendor callback triggers a separate workflow that updates the contact.
4. Store enrichment_status, enrichment_request_id, and last_enriched_at on the contact so the workflow can skip records already queued or completed.

Adding a retry inside the 20-second action makes the timeout worse; it does not add capacity or control concurrency.

Idempotency has to live outside the retry loop.

• Stable key: use the HubSpot contact ID plus a job type as the idempotency key. Send it to the vendor or store it on the queued job. If the same key arrives twice, the worker returns the existing result instead of calling the vendor again.
• State on the contact: maintain enrichment_status values such as queued, processing, complete, and failed. The enrollment criteria should exclude contacts already queued or complete.
• Disable re-enrollment for this workflow unless there is a deliberate re-enrichment policy. Re-enrollment is appropriate for a recurring refresh, not for a one-time enrichment action that can time out.
• Separate retry from duplicate submission: retries belong in the worker queue with exponential backoff. The HubSpot action should submit once and exit.
• Handle 429 explicitly: the worker should read the Retry-After header and pause the queue rather than immediately retrying the same request.

This also makes failures observable: a failed job is a queue record with a reason, not a silent workflow error.

Reduce pressure without pretending the architecture is fixed.

• Turn off re-enrollment now; it is directly causing duplicate calls.
• Narrow enrollment criteria to new contacts with a valid business email and no existing last_enriched_at value. Exclude contacts already enriched or currently queued.
• Add a randomized delay before the custom code action to spread the 9 a.m. and 1 p.m. bursts, and cap the number of contacts entering the workflow per hour if the workflow tool allows it.
• Lower concurrency at the vendor by serializing the calls from the code action, or temporarily route only a subset of leads through enrichment.
• Add monitoring for 429 counts, timeout counts, and duplicate request IDs so the team can see whether the mitigations are working.

These measures buy time, but they do not solve the fundamental mismatch between per-record synchronous calls and a rate-limited external API.