Security Fabric fragmentation after a phased FortiOS upgrade
NorthBridge Financial runs a Security Fabric rooted on a FortiGate 600F at its Chicago headquarters. The fabric includes 14 FortiGate 200F branch firewalls, 96 FortiSwitch 448D access switches, 210 FortiAP 431F access points, FortiManager 7.0.4, FortiAnalyzer 7.0.4, and FortiAuthenticator 6.4. The group is midway through a FortiOS 6.4.12 to 7.2.6 refresh. The HQ root and two branches are on 7.2.6; the other 12 branches remain on 6.4.12. Since the upgrade, FortiAnalyzer shows intermittent gaps in switch and AP telemetry, FortiManager marks the upgraded branches as configuration out of sync, and the root FortiGate CPU jumps to 85% for several minutes every hour. RADIUS admin logins through FortiAuthenticator also fail intermittently. A change freeze begins in nine days, and the CIO wants either a safe completion plan or a rollback. The team has no lab replica of the 210-AP wireless footprint.
1. What is the most likely cause of the fabric telemetry gaps, and what containment steps would you take before changing any more firmware? Explain the evidence you would collect.
Read the worked answer
Most likely cause: mixed FortiOS versions inside one fabric plus an access-layer firmware mismatch. A 7.2.6 root aggregates telemetry from 6.4.12 branches and from FortiSwitch/AP images that may not be on the recommended release for 7.2.6. The fabric daemon can spike root CPU while processing topology and telemetry, and FortiManager can show out-of-sync when the ADOM, device model, or revision differs. RADIUS failures may be collateral from root CPU or from FortiAuthenticator integration changes, not a separate fault.
Containment before more upgrades:
- Freeze further firmware changes and capture
get system status,diagnose sys csf,diagnose debug application csfd -1on the root, FortiManager task/revision logs, FortiAnalyzer event logs, and FortiSwitch/AP firmware matrices. - Confirm FortiManager 7.0.4 and FortiAnalyzer 7.0.4 are supported with FortiOS 7.2.6; if not, plan their upgrade first.
- Temporarily reduce fabric polling/telemetry load if supported, and verify RADIUS by testing a local admin account; do not leave a single auth path.
- Compare one healthy 6.4.12 branch with an upgraded branch to isolate the change.
Evidence: CSF topology, FortiManager revision diff, FortiAnalyzer telemetry gaps, root CPU process list (diagnose sys top), FortiAuthenticator logs, and AP/switch firmware compatibility.
2. Propose a firmware and validation sequence for the remaining 12 branches and the access layer. Justify why the root and access layer are ordered as you propose.
Read the worked answer
Sequence: (1) FortiManager/FortiAnalyzer to a release that fully supports 7.2.6; (2) root FortiGate only after a maintenance window and a verified rollback image; (3) the remaining 12 branches in waves of two, starting with low-risk sites; (4) FortiSwitch and FortiAP firmware after their managing FortiGate is stable, because access-layer images must match the FortiGate recommended matrix.
Why root first, access last: the root is the fabric aggregator. Leaving it on 6.4 while branches move to 7.2 prolongs mixed-version telemetry. Once the root is stable, branches can join a known-good fabric. Access layer last because AP/switch telemetry gaps often reflect image compatibility, and changing them while the root is unstable makes correlation impossible.
Validation gates per wave: fabric status green; FortiManager config sync; FortiAnalyzer logs and telemetry for AP/switch; root CPU below baseline; RADIUS admin login success; HA failover test; branch-to-hub IPsec and routing; wireless client join and roaming; rollback checkpoint.
Caveat: if FortiManager cannot manage 7.2.6, upgrade it before any more FortiGates; otherwise you cannot stage or roll back cleanly.
3. The CIO asks whether to roll back the two upgraded branches. Give your recommendation and the criteria that would change it.
Read the worked answer
Recommendation: do not roll back the root unless there is a production outage. Rolling back the root while two branches remain on 7.2.6 recreates a mixed fabric and may not restore telemetry. Instead, stabilize the root, confirm FortiManager/Analyzer support, and complete the upgrade in controlled waves.
Roll back only if: root CPU stays above 90% and causes HA failover or admin lockout; FortiAuthenticator RADIUS failures block production administration; Fortinet support confirms a known 7.2.6 defect affecting this fabric; or FortiManager cannot sync even after ADOM/revision repair.
Criteria that would change the plan: if the two upgraded branches show application-impacting issues while the root is healthy, roll back those branches first, not the root. If the root is the problem, roll back the root and pause all upgrades. Document the decision, keep the 6.4.12 images, and set a go/no-go review before the freeze.