Monitoring Gap After a Controller Migration

Why it went unnoticed:

• The old alerting depended on SNMP polling of the on-premises controllers. After migration, those targets no longer existed or no longer reported the same objects, so alerts silently stopped firing.
• The cloud dashboard was available but not wired into the paging path. A dashboard is a view, not an alerting system unless its events are routed to on-call.
• Nobody validated the monitoring path after the migration. A migration is not complete until observability is proven end to end.

Replacement architecture:

• Single source of truth for inventory: the cloud management platform's device inventory feeds the monitoring system, so devices cannot exist in one and not the other.
• Event-driven alerting: subscribe to platform events (device offline, AP down, power loss, config change) and route them to the existing on-call/paging tool. Do not rely on polling a dashboard.
• Health polling where the platform is weak: keep lightweight polling for metrics the platform does not expose (interface errors, optic levels, power draw), targeted rather than broad.
• Synthetic checks: periodic probes from the access edge to confirm real user experience, independent of management-plane health.
• De-duplication: one alert per incident, with the platform as the authoritative device state and the monitoring system as the paging layer.

What changes:

• The SSH-to-controller workflow no longer has a valid target. Configuration changes must go through the cloud platform's API or templating model.
• The pipeline needs to authenticate to the platform (API tokens/credentials), and those credentials must be managed, rotated, and scoped.
• Change validation must move to the platform's configuration model: preview/diff before apply, and a post-apply verification step that reads back the intended state.
• The pipeline should record who changed what and when, using the platform's audit trail as the system of record.

Risks introduced:

• Credential exposure: API tokens in a pipeline are a high-value target. Use scoped tokens, secret storage, and rotation.
• Blast radius: a bad template applied through the API can hit many devices at once. Require staged rollout (canary site, then wider) with automatic rollback.
• Loss of a tested path: the old SSH workflow was known-good. The new one must be tested in a lab or a single site before broad use.
• Drift between pipeline intent and platform state: if someone changes configuration in the dashboard, the pipeline may overwrite it. Define the platform as authoritative or the pipeline as authoritative, not both.
• Dependency on external service: an outage of the cloud platform blocks changes. Document the manual fallback and its limits.

Prove monitoring:

• Inject a controlled failure in a non-production or low-impact area (e.g., power down a lab AP or block its uplink) and confirm an alert reaches on-call within the target time.
• Verify alert content includes device identity, site, and likely cause, so the responder can act without opening a dashboard.
• Confirm alert de-duplication and auto-resolution when the device returns.
• Test the paging path itself (on-call rotation, escalation) with a drill.

Prove automation:

• Run a no-op change through the pipeline and confirm preview, apply, and read-back all succeed.
• Run a canary change at one site, verify intended state, then roll back and verify rollback.
• Confirm the audit trail records the change with the correct identity and timestamp.
• Test the manual fallback path and time it, so the team knows how long it takes if the platform is unavailable.

Prove the whole chain:

• A single end-to-end test: make a change through the pipeline, break the affected device, confirm the alert fires and references the change.
• Record the results and the review date in the migration closeout document. The migration is complete when observability and change control are demonstrated, not when the cutover window closes.