Intermittent Network Collapse at a Regional Secretariat
Kurunegala Regional Secretariat supports 140 staff across three floors. Six weeks ago, a vendor installed a new Layer 2 access switch on the second floor and re-cabled 40 ports. The network uses VLAN 10 for data, VLAN 20 for IP telephony and VLAN 99 for guest Wi-Fi. A 1 Gbps fibre uplink connects to the core, and internet capacity is 300 Mbps. Since the change, staff report a daily failure between 14:00 and 15:00: database queries time out, IP phones produce one-way audio, and guest Wi-Fi slows sharply. The core switch CPU reaches 85 percent, duplicate-IP alerts appear, and no UPS alarm is recorded. The vendor recommends buying more internet bandwidth. The office has no current network diagram, and change records for the switch configuration are incomplete. The secretary wants a fix before a provincial audit in two weeks.
1. Using the symptoms, separate the likely fault domains and describe the first four evidence-gathering actions you would authorize. Explain what each action would confirm or exclude.
Read the worked answer
Fault domains. The daily 14:00-15:00 pattern and VLAN-specific symptoms point first to Layer 2 and local addressing faults, not internet capacity. A 1 Gbps uplink with 300 Mbps internet can still be congested, but 85 percent core CPU plus duplicate-IP alerts and one-way voice audio are stronger evidence of a loop, broadcast storm, rogue DHCP server, or duplex mismatch on the new second-floor switch.
First evidence actions.
- Rebuild the topology and obtain the switch configuration and change log. This confirms whether VLAN 10, 20 and 99 are trunked correctly and whether unused ports were disabled.
- Collect per-port and per-VLAN counters, broadcast/multicast rates, MAC-table flapping and STP topology from the new switch and core. This would confirm or exclude a loop and identify the offending port.
- Inspect DHCP and ARP logs for duplicate addresses and rogue servers. Duplicate-IP alerts suggest scope exhaustion or an unauthorized DHCP device.
- Correlate the 14:00 window with scheduled backups, cleaning devices, guest access patterns and voice traffic; then test latency, jitter and packet loss separately for each VLAN. This separates a time-driven load or backup problem from a persistent configuration fault.
I would not approve a bandwidth purchase until these checks are complete, because the evidence does not yet show the internet link is the bottleneck.
2. Recommend immediate containment and permanent remediation, including change-control and rollback steps. State which action you would not take yet and why.
Read the worked answer
Containment. Enable storm control, BPDU guard and root guard on access ports; disable unused ports; isolate guest Wi-Fi in VLAN 99 with rate limiting; correct any speed/duplex mismatch; and enable DHCP snooping plus dynamic ARP inspection where supported. These are reversible and reduce blast radius while evidence is collected. Do them in an approved maintenance window with a saved rollback configuration and out-of-band console access.
Permanent remediation. Produce an accurate network diagram, standard port configuration, port security, QoS for VLAN 20, monitoring alerts for broadcast rate and CPU, and a documented change process. Schedule a post-change review.
Not yet. I would not replace the core switch or mass-upgrade firmware before the fault is isolated, because that adds risk without evidence and may mask the real cause. Change control should record each step, owner, test and rollback.
3. Define three success measures and a communication approach for the secretary and audit team.
Read the worked answer
Success measures should be tied to the symptoms: zero duplicate-IP alerts; packet loss below 1 percent and voice jitter below 30 ms during 14:00-15:00; core CPU below 60 percent at peak; and guest throughput within the agreed service level. Report a before-and-after baseline, not just a post-fix snapshot.
For the secretary and audit team, I would provide a one-page incident timeline, the evidence that confirmed the root cause, approved change records, residual risks and a weekly monitoring summary. This gives assurance that the fix is controlled and repeatable rather than a temporary reset.