Monitoring That Could Not Explain a 20-Minute Outage
A 2,300-device estate (1,900 Catalyst switches, 320 APs, 80 branch routers) is polled by a single SNMP collector across a 50 Mbps management WAN link that averages 78% utilization. The collector polls 48 interface counters plus six ifXTable OIDs on every device every 30 seconds, and the same link carries syslog. NetFlow v9 is exported only from the four campus core switches at 1:1024 sampling. The NOC dashboard, built from SNMP counters, reports total campus traffic at 1.1 Gbps; the flow tool reports 28 Mbps and labels 60% of bytes as unknown. During an incident one access switch dropped 30% of its users for 20 minutes. The post-incident review found only a rate-limited syslog line and a three-minute gap in SNMP polling for that device, and nobody could say which users or applications were affected.
1. Reconcile 1.1 Gbps from SNMP against 28 Mbps from flows, and explain the 60% unknown. State which figure you would trust and why.
Read the worked answer
The flow figure is closer to reality; the SNMP number is an artefact of how it is summed.
- The dashboard sums
ifHCInOctetsandifHCOutOctetsacross all core interfaces. In a three-tier fabric a single packet crosses spine ingress, spine egress, leaf ingress and leaf egress, so it is counted four to six times. Adding both directions on top of that roughly doubles it again. A 40x overstatement is entirely consistent with multi-point counting plus direction doubling. - Check for counter wrap or a 32-bit OID being read where the 64-bit is needed - a 1 Gbps interface wraps a 32-bit counter every 34 seconds, and a 30-second poll is right on the edge of aliasing.
- Trust flows for volume, SNMP for interface state. If a single volumetric number is needed, derive it from one point (the internet edge or the WAN uplink), not from a sum across the fabric.
The 60% unknown:
- Export is configured on the core switches only, so intra-leaf east-west traffic is either unseen or seen twice with different exporters.
- The collector cannot resolve reverse DNS for internal RFC1918 space, so it buckets by address as unknown. Confirm whether unknown means "no DNS" or "no application match" - they have different fixes.
- Sampling at 1:1024 with a short flow cache can miss short-lived flows entirely, which skews the distribution even when the byte totals roughly agree.
Judgement: the dashboard is not measuring what the NOC thinks it is measuring. Fix the definition before tuning thresholds on it.
2. The access switch dropped 30% of its users for 20 minutes and the tooling could not describe the blast radius. What data sources would have answered 'who and what', and what has to change to collect them?
Read the worked answer
What would have answered it:
- Per-session syslog and traps at the source. A switch that loses 30% of its ports emits a burst of link-state and authentication messages. Rate limiting silently discarded the unique ones. Keep rate limits, but exempt state-change and security severities, and rely on the device's local buffer (
show logging) as the capture of last resort - 20 minutes of history is almost always still in the buffer when the ticket is opened. - Missing-poll as an alarm. A three-minute poll gap is a signal, not a nuisance. The collector should alarm on the absence of data at the expected interval, per device, before any threshold is evaluated. Most outages announce themselves as missing data first.
- Identity. Port-level 802.1X/MAB session data (or the ISE/DNA-C client table) converts "30% of ports down" into "these 40 named users and these 12 MACs". Without it, the review can only report a port count.
- Per-interface state at higher fidelity than 30-second polling - traps carry the transition time; polling only tells you the state at the sample instant and can miss a flap that healed between polls.
What has to change: a management-plane design, not more tools. Decide which data needs to be immediate (state change, authentication failure, PoE fault), which can be periodic (utilization), and which can be streamed on change (interface counters, via model-driven telemetry) - and make sure the path carrying that data is not the same saturated link carrying syslog.
3. The collector link is at 78% of 50 Mbps. Do the arithmetic on the polling load and propose a design that fits.
Read the worked answer
The arithmetic:
1,900 devices x 48 interfaces x 6 OIDs / 30 s = roughly 18,200 request/response pairs per second. At an average of ~250 bytes per response, including SNMP, UDP and IP overhead, that is on the order of 35–40 Mbps of the 50 Mbps link - which is exactly the observed 78%. Polling 48 access ports per switch to learn that they are mostly idle is the single biggest driver.
Design that fits, in order of return:
- Poll what you act on. For a 48-port access switch, four uplink counters matter continuously; the 44 access ports need state-change detection, not byte counts. That alone cuts the poll volume by roughly 90%.
- Differentiate intervals. 30 s for uplinks, core and WAN; 5 minutes for access ports; event-driven traps for state.
- Use model-driven telemetry for the high-rate data. gNMI/streaming at a controlled cadence sends only what changed and removes the request/response overhead entirely - but it also multiplies potential volume if left with default subscriptions, so set the cadence deliberately.
- Separate the paths. Syslog, SNMP and telemetry on the same constrained WAN link means a poll storm degrades logging. A dedicated management VRF with a low-rate path, or local collectors at large sites that forward summaries, removes the shared fate.
- Collector-side: confirm it is not walking large tables (ARP, CDP neighbours, full ifDescr) on every cycle; those are the operations that produce megabyte-scale bursts rather than steady load.
Judgement: the constraint is the 50 Mbps link, not the collector's capability. Any design that keeps 30-second full-interface polling across 1,900 devices will fail again the moment the estate grows, so fix the polling model rather than buying a bigger circuit.