A 22-Minute Polling Job, a Leaked Key, and a Deadline for Alerting

Two independent causes, and the fix needs both.

Cause 1: too many calls. Per-network, per-endpoint iteration means roughly 800+ calls per run, and the client list is the expensive one. Four hundred networks times two endpoints, every fifteen minutes, is a request volume designed to hit the organization-scoped rate limit - which is why the 429s appear. The 429s then feed the second cause.

Cause 2: the retry logic amplifies the problem. Immediate retry with no backoff on a 429 is a self-inflicted denial of service: the throttled client retries instantly, consumes more of the same budget, gets throttled harder, and the run stretches from minutes to tens of minutes. That is exactly how a 15-minute job becomes a 22-minute one and leaves inventory stale for 40.

Redesign:

• Honor the rate limit explicitly. Cap concurrency, and on a 429 read the Retry-After header and sleep for that duration. Add exponential backoff with jitter for 5xx errors.
• Cut call count. Use per_page and follow the pagination Link header rather than paging manually, and pull org-level inventory once instead of per network. Device inventory changes on a scale of days, not minutes - pull it hourly or on a webhook, not every fifteen minutes.
• Tier the data by how fast it actually changes. Uplink status every minute; client lists you probably do not need at all for this summary.
• Alert on staleness itself, so a frozen job pages someone instead of silently serving old inventory.

Webhooks, with polling as a backstop. Uplink failure is an event the organization already knows about; polling for it means your detection latency is bounded by your poll interval, and tightening the interval to seconds across 400 networks is exactly what put the current job over its budget. Meraki webhooks push alert events, including uplink and device availability alerts, to an HTTPS endpoint you control.

The operational catches, in the order they will bite:

• You need a public, TLS-terminated receiver that can absorb bursts and return quickly. If the receiver is slow or down, you lose events silently.
• Validate the payload. Webhooks carry a shared secret for signature verification; an unauthenticated receiver is an open endpoint anyone can post fake outages to.
• Treat webhooks as at-least-once, not exactly-once. Deduplicate on event identity, and make the alert path idempotent so a duplicate does not page twice.
• Keep the backstop. Webhooks can be dropped. A coarse poll - a few minutes - that compares against last-known state catches the events the receiver missed, and catches the case where the receiver itself died.
• Alert on alerting. If no webhook has arrived in 24 hours, that is itself a signal worth a ticket.

What it exposes: a Meraki API key is issued to a dashboard administrator and carries that administrator's privileges across every network and organization they can reach. There is no separate permissions model for keys themselves - the key is the admin, minus the password. Here that is four organizations' worth of read and, depending on that admin's role, write access to every network in the organization. Anyone holding it can enumerate the entire estate, read client and device data, change configuration, and - importantly - do so without leaving a password-login trace that would look anomalous.

Remediation sequence matters:

1. Revoke first, then think. Disable or delete the departed admin's dashboard account; that immediately invalidates the key. Do this before the redesign work, not after.
2. Confirm the job did not quietly break - the script you just fixed will start returning 401s, and you want that to be a planned step rather than a 3 a.m. discovery.
3. Issue a replacement key under a purpose-built service account with the narrowest role that covers the job, and store it in a secrets manager rather than in the cron VM's environment or, worse, in the script.
4. Audit the change log for that admin identity over the period since departure, looking for writes that did not come from your automation.
5. Fix the root cause: a joiner-mover-leaver process that includes API keys and webhook secrets, not just accounts. A key outliving its owner by a month is not a scripting problem, it is an offboarding gap that happened to be found by a scripting problem.