Four Hours of Silence
At 02:14 the ERP cluster failed over to the standby data centre after a leaf switch pair in the primary site flapped 40 times in 25 minutes. About 200 VMs saw up to 12% packet loss while ECMP and OSPF reconverged repeatedly. Nobody was paged. The NMS polls interface counters every five minutes, alarms only on 'link down', and SNMP traps from that site are rate-limited to 20 per minute at the collector, with the excess dropped. The first human awareness came from a user ticket at 06:20. Post-incident data shows a fibre SFP on uplink 2 with CRC errors rising from 0 to 9,400 in an hour, and one power supply in leaf B logging a 3-second brownout. A second, shorter flap occurred three days later.
1. Reconstruct the likely failure chain and specify the evidence you would pull to confirm or reject it.
Read the worked answer
Chain: power supply brownout on leaf B → line card or SFP reset → CRC errors climb on uplink 2 → interface flaps → OSPF adjacency resets and ECMP withdraws the path → asymmetric and lossy forwarding for ~200 VMs → ERP heartbeat timeout → cluster failover at 02:14.
The competing hypothesis is that the SFP degraded first and the brownout is coincidence. The evidence decides which:
- Timing correlation: syslog and BMC/IPMI power-event logs, all NTP-synced. If the brownout precedes the first CRC by seconds, the PSU is causal.
- Error trend:
show interfaces counters errorsandshow controllerson uplink 2 across the window - CRC rising monotonically before the first flap points at the optic; CRC appearing only after flaps points at the reset. - Control plane: OSPF adjacency logs and
show ip ospf neighborhistory; verify reconvergence count matches the 40 flaps. - Environmental:
show environment/ PSU history on leaf B, and whether leaf A's PSUs show the same input feed.
If both the SFP and the PSU are implicated, replace both - do not ship a fix that only addresses the optic.
2. Why did monitoring miss it, and what specifically would you change?
Read the worked answer
Five gaps, each independently sufficient to cause the silence:
- Five-minute polling cannot see sub-minute flaps or a 40-flap burst; a counter that is normal at every sample reads as healthy.
- Alarming only on 'link down' ignores CRC errors, flap counters and adjacency changes - the leading indicators here.
- Trap rate-limiting dropped exactly the traps generated by the storm, and nothing alarmed on dropped traps.
- No symptom-level probe: no synthetic ERP transaction from both sites, so 12% loss on 200 VMs was invisible.
Changes:
- Alarm on error deltas per interval (any CRC increment, not a threshold of 9,400), using streaming telemetry or 30-second polling on uplinks.
- Alarm on interface flap count and on OSPF/BGP adjacency transitions.
- Raise the collector's trap limit, and add a meta-alarm on trap drops and on 'no data from device'.
- Synthetic transaction every 60 seconds against ERP from each site, alerting on loss and latency not just reachability.
- Route that alert class to the network on-call, not to a queue.
3. What do you do in the next 24 hours versus the next 30 days, and what would you refuse to do?
Read the worked answer
Next 24 hours: replace the SFP and its patch lead, reseat and test the leaf B power supply on the alternate feed, keep ERP pinned to the standby site until primary is stable for 24 hours, verify NTP on all network devices, enable adjacency-change and CRC-delta alerts, and have a named engineer watch the next maintenance window.
Next 30 days: redesign telemetry for uplinks (streaming or 30-second), add flap dampening and root-cause the second flap at day three, review power redundancy per rack so one brownout cannot take a leaf pair, write the runbook, and run a game day that injects a flap to prove someone is paged inside five minutes.
Refuse: disabling ECMP or pinning traffic to one uplink - that hides the fault and creates a worse one; raising ERP heartbeat and failover timeouts so the symptom disappears; accepting 'failover worked, so nothing to fix' from the application team; and approving a monitoring-platform replacement before the existing thresholds and trap handling are fixed. Buying a new tool to solve a configuration problem is how you get the same silence on better hardware.