Rate Limits and Async Tasks in a Catalyst Center Integration
An integration polls a Catalyst Center cluster every 5 minutes to populate a client-health dashboard for 1,800 access points across 60 sites. The script calls /dna/intent/api/v1/client-health once per site and then calls the task endpoint for each command-runner job it triggers. After roughly 40 calls the cluster returns HTTP 429, and sometimes the Retry-After header is missing. Authentication uses a token with a 3,600-second lifetime; long runs fail with 401 and the script re-authenticates inside the loop. Pagination defaults to 500 records and the script currently ignores totalCount. Two worker nodes run the same job for high availability, so duplicate polls and duplicate command jobs reach the cluster. The business needs 15-minute freshness, not real time, and the cluster is shared with the operations team during business hours.
1. Redesign the polling approach to respect rate limits while meeting the 15-minute freshness requirement.
Read the worked answer
Freshness is 15 minutes, so 5-minute polling is over-engineered and is a direct cause of the 429s.
- Poll once per 15 minutes, not once per site. Read the site hierarchy once to obtain site IDs, then use a bulk client-health endpoint with pagination (
limit=500, followtotalCount) instead of 60 per-site calls. - Centralize the schedule on one leader (see the HA question) and cache results locally so dashboard reads never hit the API.
- Honor
Retry-Afterwhen present; when absent, use exponential backoff with jitter, for example 1, 2, 4, 8, 16 s capped at 60 s, plus a circuit breaker after repeated 429s. - Spread reads across the 15-minute window with a token bucket instead of a thundering herd at :00.
- Request only the fields the dashboard needs to reduce payload and processing time.
Measure calls per interval and pages per run. If you are still near the limit, reduce frequency or move to an event or notification mechanism rather than increasing the rate. The dashboard's 15-minute freshness budget is the design constraint; the API limit is not negotiable.
2. How should the integration handle token lifetime and async task polling without producing duplicate or orphaned work?
Read the worked answer
Token handling: refresh proactively at about 80% of the 3,600 s lifetime, not reactively inside a loop. Store the token with an expiry timestamp and make the HTTP client single-flight so two threads do not both re-authenticate. On 401, refresh once and replay the request idempotently; if the replay also fails, abort with a clear error. Never re-authenticate in the middle of a pagination loop without resetting the page cursor, or you will duplicate pages.
Task lifecycle: command-runner endpoints return a taskId. Poll getTaskById with exponential backoff starting at 2 s, capped at 30 s, and a hard timeout such as 10 minutes. Treat progress endpoints as advisory. On timeout, stop polling, record the task ID for support, and do not resubmit blindly; first check whether the task actually completed. Distinguish 404 (task expired or unknown) from 5xx (retryable) in the error handler.
Idempotency: derive a key from device ID, command set, and intended-change hash. Record it locally before submitting and check both the ledger and the task status after a restart. That is what prevents orphaned command jobs from being submitted twice.
3. Two worker nodes run the same job for high availability. What design avoids duplicate polling and duplicate command jobs?
Read the worked answer
Two independent pollers double the request rate and duplicate command jobs, which is both a rate-limit problem and a correctness problem.
Options:
- Leader election with a lease: use a distributed lock (Redis, a database row, or a Kubernetes Lease) with a TTL. Only the leader polls and submits jobs; the standby takes over when the lease expires. This is simple and sufficient for 15-minute freshness.
- Sharding by site: each worker owns a disjoint site list with a shared cursor. This scales better but complicates failover and rebalancing.
For command jobs, add an idempotency key derived from (deviceId, command, intendedChangeHash) and persist it in a ledger before submission. On restart or failover, check the ledger and the task status instead of resubmitting. Also schedule heavy reads outside business hours where possible, and cap concurrent requests per worker so one node cannot consume the whole budget during a failover.