APIC Inventory at 6,400 EPGs
Your team runs three APICs in a production ACI fabric with 6,400 EPGs across 180 tenants. A Python inventory job authenticates to /api/aaaLogin.json every time it fetches a page, then calls /api/class/fvAEPg.json?page-size=500&page=0..N until imdata is empty. It runs every 60 seconds from four parallel worker processes. Over the last week APIC CPU has risen to 85%, the job intermittently gets HTTP 429 with Retry-After, and the resulting CMDB is missing 300-900 EPGs per run. The APIC audit log shows thousands of successful logins per hour. The fabric has no APIC fallback configured for the script. You must restore reliable inventory and prepare for 5-second EPG change detection without adding APIC load.
1. The script retrieves only part of the EPG inventory and sometimes publishes an incomplete CMDB. Explain the likely causes and the correct REST retrieval pattern, including how to use APIC pagination metadata.
Read the worked answer
The missing EPGs are a pagination and retry bug, not an APIC capacity limit. APIC returns the first page with totalCount; the loop must use that count and the accumulated imdata length. With page-size=500, 6,400 EPGs need 13 pages (0-12). If a 429 occurs, the current code may treat the error body as an empty page and stop early. The thousands of logins are caused by re-authenticating per page per worker. Fix: authenticate once per worker, reuse the APIC cookie/token, and refresh only on 401/403 or before token expiry. Loop until accumulated objects equal totalCount, not until an empty page. Validate each run by comparing totalCount to the CMDB insert count. Add exponential backoff with jitter and honor Retry-After; reduce four workers to one or two while the bug is fixed. Log page number, HTTP status, and object count. A higher page size reduces request count but increases response size; 500 is reasonable. Do not use four workers to fetch the same class concurrently; partition by tenant if parallelism is required.
2. Design a change to detect EPG creation within 5 seconds without polling every EPG. Specify the APIC subscription mechanism, scope, refresh strategy, and failure recovery.
Read the worked answer
For 5-second detection, polling cannot meet the goal without 12x the current request rate. Use an APIC WebSocket subscription. Prefer one class-level subscription to /api/class/fvAEPg.json?subscription=yes rather than 6,400 per-object subscriptions; APIC sends the initial snapshot and then change events. Set a refresh-timeout (for example 60 seconds) and send refreshSubscription before it expires. Authenticate once, open the WebSocket with the session token, and handle 401/403 by re-authenticating and resubscribing. The initial snapshot can be large; page or filter the initial inventory separately if needed, but the subscription stream should be one connection. If APIC limits class-level subscription size, partition by tenant (180 subscriptions) rather than per EPG. Use a queue between the WebSocket handler and CMDB writer so a slow database does not drop events. On websocket close, reconnect with backoff and perform a full reconciliation query, because events may have been missed. Do not run the subscription from all four workers; one active subscriber per fabric with a standby.
3. What controls would you add so the inventory job stops triggering 429 responses and never silently publishes partial data again?
Read the worked answer
Preventing 429 and missing data: (1) backoff with Retry-After and capped retries; (2) one login per worker, not per page; (3) completeness check against totalCount and fail the run rather than publishing partial data; (4) reduce concurrency and stagger runs; (5) move steady-state inventory to the subscription and use REST only for reconciliation; (6) use a dedicated APIC service account with read-only role and least privilege. Alert if APIC CPU > 80%, 429 rate > 0, or login count/hour exceeds a baseline. If APIC fallback is required, configure the script to try the standby APIC only on connection failure, not to spread load. Test with a canary tenant before enabling all 180. This addresses load and completeness without changing the fabric.