Event-Driven Integration with ServiceNow and Slack
A managed service provider runs one Mist organization with 1,800 APs and 300 switches across 90 customer sites. It uses ServiceNow for incidents and Slack for NOC alerts. Today a polling script calls the Mist API every 60 seconds for each site to detect AP down and switch port down events. The script makes about 12 calls per site per minute, hits HTTP 429 during peak hours, and still misses short outages. The provider wants near-real-time incident creation, enrichment with site and device details, and duplicate suppression. Peak event volume is 400 webhook events per hour; normal is 40. The security team requires HTTPS, authenticated delivery, and no long-lived org-wide API token in the polling script. A new integration service can be deployed in the provider's cloud.
1. Design the event flow using Mist webhooks and REST API. Which webhook topics and filters should be used?
Read the worked answer
Replace polling with Mist webhooks delivered to the new integration service. Subscribe to device-updowns for AP and switch state changes, and to alarms for conditions such as PoE budget, bad cable, or port flap. Do not subscribe to every topic; unnecessary events increase cost and duplicate incidents. If the webhook configuration supports site or device-type filtering, scope it to the 90 customer sites and the relevant device types. The integration service receives the event, validates it, and calls the Mist REST API only when enrichment is needed: site name, site address, device name, model, MAC, and current status. Create the ServiceNow incident from the enriched event and post a concise Slack alert for severity-one sites. This gives near-real-time detection and removes the per-site polling loop that causes 429 responses.
2. How do you meet the security and duplicate-suppression requirements?
Read the worked answer
Terminate the webhook on an HTTPS endpoint with a valid certificate and restrict inbound traffic to the provider's integration service. Use the webhook secret or signature mechanism supported by Mist, validate it on every request, and reject unsigned or stale deliveries. Do not embed a long-lived org-wide API token in the polling script; instead, create a dedicated integration service identity with the least privilege needed for site and device read access, store the credential in a secrets manager, and rotate it. For duplicate suppression, derive an idempotency key from the event identifier and device/site identifiers, store recent keys for a defined window, and check ServiceNow for an open incident on the same device before creating another. If a duplicate event arrives, update the existing incident instead of opening a new one.
3. What API calls and rate-limit handling would you use for enrichment without causing 429s?
Read the worked answer
Use targeted read calls: retrieve site details by site ID, and retrieve device details or statistics for the specific device rather than listing all devices at every site. Cache site metadata and device inventory for a short TTL because it changes infrequently, and invalidate the cache on device-updown events. Batch enrichment where the API supports it, and queue webhook events so the integration service controls concurrency. On HTTP 429, honor Retry-After if present and apply exponential backoff with jitter; do not retry immediately in a tight loop. Track 429 rates and event lag as service metrics. Because peak volume is only 400 events per hour, a modest queue and cache should keep API usage far below the old polling design, while preserving near-real-time incident creation.