Four Thousand Alerts and One Missed Outage
A network team supports 400 switches and routers. Syslog goes to a collector, SNMP polls every 60 seconds into a monitoring server, and every alert lands in one shared mailbox. The team receives about 4,000 alerts a day and has stopped reading them. Last month a distribution-switch outage went unnoticed for 40 minutes; afterward, syslog entries from the three devices involved appeared out of order, and a certificate on the management jump host had expired 11 days earlier with no warning. Time on network devices comes from an internal NTP server that syncs to an unreliable public source; core switches drift up to 90 seconds. Change records live in a shared spreadsheet anyone can edit.
1. Rank the operational failures here and state what you would fix first, with justification.
Read the worked answer
The alerting model is the root cause, but time sync is the cheapest fix and a prerequisite for everything else.
Ranking:
- Time synchronization. Fix first because it is a small, low-risk change that makes every other signal trustworthy. You cannot correlate three devices' logs, order an incident timeline, or trust a certificate check while clocks drift 90 seconds apart.
- Alert design - volume, not detection. The team has the data; it has lost the ability to distinguish signal. 4,000/day is roughly 6 alerts per device per day, which means thresholds are set at "any counter moved" rather than "service is affected."
- Dependency suppression. A single upstream failure generates alerts from every downstream device. Without parent/child suppression, one outage becomes hundreds of alerts and the real one is buried.
- Change management as a spreadsheet. Untracked changes make correlation impossible, because a config change is indistinguishable from a fault.
Why not start with alert design? Because alert tuning depends on being able to trust timestamps and to know whether a device rebooted before or after a change. Do the NTP fix in a day, then tune.
2. Explain concretely how the 90-second clock drift degrades troubleshooting, monitoring, and security, and how you would design the NTP service.
Read the worked answer
Troubleshooting. Syslog entries from device A at 10:14:05 and device B at 10:13:40 may be the same event or two events a minute apart. Order-of-events reasoning - did the link flap before the OSPF adjacency dropped? - collapses. Post-incident timelines become guesswork.
Monitoring. SNMP trap timestamps disagree with the poller's clock, so the monitoring server files traps under the wrong interval or discards them as out-of-order. Correlation rules that look for "B within 30 s of A" fire correctly or not depending on which device's clock you believe.
Security. Certificate validation and Kerberos/802.1X tolerate only small skew (Kerberos default 5 minutes). Drift also makes log-based forensics and any audit trail unreliable, and hides events just outside a retention window.
NTP design:
- Run at least three internal NTP servers at stratum 2, syncing to two or more independent, authenticated upstream sources; never a single unreliable public server.
- Point all network devices at the internal servers by IP, with an ACL restricting NTP to infrastructure.
- Use authentication (symmetric keys or NTS where supported) so a spoofed source cannot move the clock.
- Monitor offset per device and alert above a small threshold (e.g. 500 ms) - drift is itself a signal of a device that has lost its config or is overloaded.
- Do not chase sub-millisecond precision; consistency matters far more than absolute accuracy here.
3. Propose a replacement operational model for alerting and for change control that would have caught this outage and the expired certificate.
Read the worked answer
Alerting
- Alert on symptoms, not counters. Page on "site unreachable," "routing adjacency down > 2 min," "HA pair member lost," not on a 60-second poll of interface utilization.
- Tier the output: page (human must act now), ticket (act in business hours), dashboard only (trend). Anything that does not fit a tier gets deleted, not muted - muted alerts rot and then get ignored.
- Dependency suppression and flap damping: suppress child alerts when the parent is down; require N consecutive polls before alerting.
- Maintenance windows tied to change records so planned work does not train people to ignore pages.
- Active/synthetic checks from two vantage points: the monitoring server already sits inside the network, so it cannot prove the network is reachable from the business side.
Certificate and configuration hygiene
- Inventory every certificate with expiry dates and owners; alert at 60/30/7 days to a named person, not a shared mailbox.
- Nightly automated config backups with diffing - an unplanned diff is often the leading indicator of an outage.
Change control
- Move from the spreadsheet to a ticketing system with required fields: devices, rollback plan, window, and validation. Tie each change to the monitoring maintenance window automatically so the two cannot drift apart.
- Post-incident, verify the fix was recorded as a change; if the outage was caused by an undocumented change, that is the finding, not the switch failure.