Intermittent Outages on a Two-Member LACP Bundle
A firewall connects to a pair of core switches through a two-member 10G LACP bundle, one member to each switch in an MLAG pair. VLANs 10 (users), 20 (voice), 30 (storage, jumbo frames), and 99 (management) cross the bundle. VLAN 99 is configured untagged on the firewall, while the switch ports have native VLAN 1. Users report 30-90 second outages several times an hour, worst during the backup window. Switch logs show rapid spanning-tree topology changes and LACP member 2 cycling every 10-20 minutes; the firewall logs member 2 as down with no optics alarm. Storage replication over VLAN 30 fails on large writes while small pings succeed. An external uptime monitor on VLAN 10 has recorded no outages at all.
1. Separate the symptoms into distinct faults. In what order would you investigate and remediate them, and why?
Read the worked answer
There are at least three independent faults; treating them as one will produce a guessing loop.
Fault A - unstable bundle member. Member 2 cycling every 10-20 minutes with no optics alarm is consistent with a unidirectional or marginal physical link (bad optic, dirty connector, or a link that comes up but does not pass traffic), or with an LACP timer/configuration mismatch between the two switch peers. Because one member keeps leaving and rejoining, spanning-tree reconverges and the hashing bucket set changes, producing the 30-90 second outages. The uptime monitor sees nothing because its flow hashes to member 1 the whole time - it is monitoring one path, not the bundle.
Fault B - native VLAN mismatch. The firewall sends VLAN 99 untagged; the switch expects untagged frames in VLAN 1. Management frames are therefore placed in VLAN 1, and any untagged traffic in VLAN 1 leaks into VLAN 99 on the firewall side. That can create a bridging loop or a broadcast domain merge, which also produces topology-change churn and intermittent loss.
Fault C - MTU/jumbo mismatch on VLAN 30. Large writes fail while pings succeed: classic size-dependent failure where one side is at 9000 and the other at 1500, or where TCP MSS is not adjusted for jumbo.
Order: stabilize the physical/layer-2 bundle first (Fault A), because nothing you measure is trustworthy while a member is flapping; then fix tagging (Fault B), which is a correctness and security issue; then address MTU (Fault C), which is isolated to storage and can be scheduled. Remediating MTU first would be invisible under the flapping.
2. What is the correct VLAN tagging fix, and what is the operational risk of leaving it as-is?
Read the worked answer
Correct fix: make the untagged/native VLAN consistent end to end. Either (a) configure the switch ports' native VLAN to 99 and keep 99 untagged on the firewall's subinterface, or (b) better, tag VLAN 99 on both sides and set the switch native VLAN to an unused, blackholed VLAN (not VLAN 1), so no untagged frame is ever mapped into a live segment.
Whichever you choose, verify: the firewall's trunk allows exactly 10, 20, 30, 99 and no more; the switch native VLAN matches on both MLAG peers (a mismatch between the two peers causes loops when a member flaps); and management access is retested from a host that is not directly attached.
Risk of leaving it: untagged frames from the firewall's VLAN 99 land in switch VLAN 1, so management traffic is bridged into the user VLAN. That is both a segmentation failure (management plane reachable from a user segment) and an availability risk: a loop or broadcast storm across the mismatched native VLAN can take down the bundle, and the STP churn you are already seeing may partly be caused by it. Debugging is also harder because the same symptom is produced by two different faults.
3. How would you handle jumbo frames for VLAN 30, and would you enable jumbo on the firewall interface at all?
Read the worked answer
Jumbo frames must be consistent on every hop and every device on the path, or not used at all. Steps:
- Confirm the storage endpoints, both switches, and both MLAG peers support and are configured for 9000-byte MTU on VLAN 30; a single 1500-byte hop turns the whole path into a black hole for large frames.
- Decide per VLAN, not per physical interface. Most firewalls allow per-subinterface MTU or per-zone MTU; set VLAN 30 to 9000 only if the platform supports it and the firewall is actually routing that storage traffic. If storage traffic should not traverse the firewall at all, the right answer is to keep it layer 2 and leave the firewall at 1500.
- If the firewall must route VLAN 30 at 1500 while hosts use 9000, enable TCP MSS clamping on that VLAN so TCP connections negotiate a size that fits. That fixes TCP but not UDP-based replication or NFS over UDP.
- Validate with DF-set pings at 8972 and with a real large write, and check interface counters for giants and drops.
Judgment: enabling jumbo on a firewall is a liability unless there is a measured throughput need. Prefer keeping storage on the switching layer; if it must cross the firewall, enable jumbo end to end and monitor for giant/drop counters rather than assuming it works because pings pass.