429s, Vanishing Webhooks, and an Analyst With an MCP Token
Cobalt Systems (400 employees) built an internal service called OpsHub that mirrors 12 ClickUp Lists into its data warehouse and writes status changes back. The sync runs every 15 minutes and re-reads every task in the 12 Lists - about 9,000 tasks and 40 Custom Fields each time. Within a week the job mostly receives 429 responses and a single pass takes 22 minutes. An engineer adds webhooks as a fix. The OpsHub endpoint performs a database write and a Slack call before responding, taking 12 seconds, and webhooks begin "disappearing". Separately, a data analyst has connected an AI client to ClickUp through the MCP server using a personal API token, so they can ask questions and create tasks. Security found the token in the analyst's shell history and wants a review before it is used again.
1. Redesign the polling sync. What specifically causes the 429s, and what is the right architecture?
Read the worked answer
The 429s are a volume problem with a shape problem underneath. Reading 9,000 tasks with 40 fields every 15 minutes is roughly 96,000 task reads and millions of field values per hour, most of which have not changed. API rate limits are set per plan and per token, so the job is pushing against a ceiling it can never win.
What to change:
- Paginate properly. Task retrieval returns pages; a naive client that ignores pagination or requests oversized pages issues far more calls than needed.
- Stop reading everything. Filter by an updated-since timestamp so each pass fetches only what changed since the last successful run.
- Back off correctly. On a 429, honour the retry hint and use exponential backoff with jitter. Retrying immediately turns a throttle into an outage.
- Trim the payload. Request only the fields OpsHub actually stores.
Target architecture: webhooks drive near-real-time updates for the events that matter (task created, updated, status changed, comment added); a slower scheduled reconciliation - hourly or nightly - closes any gaps with a full compare. Keep the reconciliation even after webhooks are healthy, because it is your only defence against missed events.
Also separate the read token from the write token so a runaway read cannot starve the write-back path.
2. Why are webhooks disappearing, and how do you make delivery reliable without losing events?
Read the worked answer
The 12-second response is the cause. Webhook senders expect a fast acknowledgement - on the order of a few seconds - and treat a slow or non-2xx response as a delivery failure. They retry, then eventually mark the endpoint unhealthy and stop or disable the webhook. From OpsHub's side, events simply vanish.
Fix the handler, not the sender:
- Acknowledge immediately. The endpoint should validate the request, write the raw payload to a queue, and return a 2xx in well under a second. The database write and Slack call move to a background worker.
- Be idempotent. Retries mean duplicates. Key processing on the event and task identifiers so a repeat delivery is a no-op.
- Verify authenticity if your workspace's webhook signing option is enabled - check the signature header against the shared secret and reject mismatches.
- Detect silence. Track events received per hour; a sudden drop is your early warning that the webhook was disabled or the secret rotated.
- Reconcile. The scheduled diff catches anything genuinely lost.
Secret handling: the webhook secret and the API token belong in a secret manager, never in code or in an engineer's shell history. Rotate both as part of this work, and document the re-registration procedure so a disabled webhook is a five-minute fix, not an incident.
3. Security asks what to do about the analyst's MCP connection. What is your recommendation and the reasoning behind it?
Read the worked answer
The problem is not MCP - it is that an AI client is authenticated as a human, with a long-lived credential sitting in plaintext.
Why the current setup fails review:
- A personal API token carries that person's full permissions across everything they can see, not just the Lists the analysis needs. It cannot be scoped down.
- It is tied to a person. When the analyst changes team or leaves, the connection either dies silently or keeps working under a departed identity - the same failure mode as a shared password.
- It lives in shell history, so it is already exposed to anyone with access to that machine or its backups.
Recommendation:
- Revoke the token now and confirm what it touched.
- Reconnect through OAuth if the MCP client supports it, so authorisation is revocable and visible; otherwise create a dedicated service account with access only to the Spaces and Lists the analysis needs.
- Do not use a personal account for machine access, ever.
- Store the credential in a secret manager, rotate on a schedule, and remove it from shell history and any scripts.
- Review which tools the MCP connection exposes. Read-only query access is a very different risk from create-and-update access; grant write only if the workflow truly requires it.
- Log and review calls for the first month, then set a quarterly access review.
The governing principle: AI clients are integrations, and integrations get their own identity and least privilege.