vManage API Polling and Template Push Under Load
A managed service provider operates 650 Cisco SD-WAN edges across 14 tenants on vManage 20.12. A Python dashboard polls /dataservice/device every 60 seconds and /dataservice/device/interface every 5 minutes, then writes to a local database. After adding 200 edges, the dashboard receives HTTP 429 responses and shows stale data; some requests return only the first page. The provider also uses the same service account to push a device template to 40 new edges. One push returns 200 but only 27 edges are configured; another push duplicates a subinterface because the operator retried after a timeout. The API account has admin role because the previous engineer could not get the template API to work otherwise. The provider needs near-real-time alerting for SLA reporting but cannot add more vManage CPU.
1. Explain the 429 responses, the partial first-page results, and how a template push can return 200 while only 27 of 40 edges are configured.
Read the worked answer
429 means the client is exceeding vManage's API rate or concurrency limits, not that the credentials are wrong. Polling 650 devices every 60 seconds plus interface calls every 5 minutes creates sustained request pressure, and the addition of 200 edges pushed it over the limit. Immediate retries without backoff make it worse. The dashboard must honor Retry-After if present, use exponential backoff with jitter, and reduce request volume.
First-page-only results are a pagination bug. vManage list endpoints return a page and metadata; the client must request subsequent pages until exhausted. A response with 200 and a subset of records is a successful page, not a complete inventory. The dashboard likely ignores pageNo, pageSize, or the total count and writes only the first page to its database.
A 200 on template push means the request was accepted, not that all devices converged. Template deployment is asynchronous: the API returns a job or action identifier, and the client must poll the action status and then verify each device. Only 27 of 40 configured because the job partially failed, variables were missing for some devices, or the client never waited for completion. The duplicate subinterface happened because a timeout was treated as failure and the operator resubmitted a non-idempotent push. On retry, query the job status and device state before resubmitting.
2. Design a monitoring and alerting approach that provides near-real-time SLA data without adding vManage CPU. Include pagination, caching, and event-driven options.
Read the worked answer
Separate inventory polling from event alerting. Inventory does not need 60-second freshness; poll /dataservice/device every 5 to 15 minutes with pagination and conditional requests where supported. Cache the results and serve the dashboard from the cache. For interfaces, poll only the tenants and devices that changed, or use a longer interval and delta queries based on last-updated timestamps. Never let each dashboard user trigger independent vManage calls; centralize the collector.
Use event-driven delivery for SLA alerts. Subscribe to vManage alarm or event notifications via webhook to the provider's collector, so link-down and SLA-impacting alarms arrive without polling. If webhooks are not available for the needed event, poll the alarm endpoint with a small page size and a lastUpdated watermark, not the full device table. Use streaming telemetry from the edges to a collector for performance metrics if the platform supports it; that moves load off vManage.
Implement pagination and backoff correctly: loop until all pages are read, persist the high-water mark, honor 429 with Retry-After, and cap concurrency. Use one authenticated session per collector rather than logging in per request, and monitor the collector's own request rate. This gives near-real-time alerting for events while keeping routine polling modest.
3. Fix the template-push process and the account design. Address idempotency, timeout handling, verification, and least privilege.
Read the worked answer
Make the push verifiable and idempotent. Before attaching a template, resolve the template ID and device IDs, validate every variable for all 40 devices, and record the intended state. Submit the attach as a job, capture the job/action ID, and poll until terminal state. Then verify each device's configured template and the specific subinterface, not just the job status. If a timeout occurs, do not resubmit blindly; query the job status and device state first. If the platform has no idempotency key, implement a client-side operation record keyed by device, template version, and variable set so a retry can detect a completed or in-progress push.
Handle partial success explicitly. The 13 unconfigured devices should be retried individually after the cause is known, not by rerunning all 40. If the template recreates a subinterface, use a template that expresses the full desired state and check for existing subinterfaces before retry, or use a replace-style operation where supported.
Apply least privilege. The admin account is unnecessary and dangerous. Create a role for the dashboard with read-only monitoring and inventory access, and a separate role for template operations limited to the required tenants and templates. Do not share the dashboard's read account with the push automation. Rotate credentials, audit API use, and ensure the push account cannot change AAA, certificates, or other tenants' templates.