Firmware Upgrade Window Causes Outage
An operations team schedules a controller firmware upgrade during a Saturday maintenance window (02:00–05:00). The controller is an HA pair. The team upgrades the standby first, then the active, but does not verify synchronization between steps. On Monday, users report authentication failures and some APs are offline. Logs show the two controllers ended on different firmware versions and several APs failed to rejoin. The customer requires documented rollback and minimal disruption.
1. Identify the process failures and describe the correct HA upgrade sequence with rollback.
Read the worked answer
Process failures:
- Upgrading the active before confirming the standby was fully synchronized and healthy.
- No version-parity check between steps.
- No post-upgrade verification or rollback trigger.
Correct sequence:
- Back up configuration and record current versions.
- Upgrade the standby, then verify it boots, syncs, and is healthy.
- Fail over to the upgraded standby and confirm clients/APs are stable.
- Upgrade the former active, verify sync and version parity.
- Post-upgrade validation (AP joins, authentication, client connectivity).
- Define rollback: if validation fails, revert to the prior image on both controllers using the documented procedure and restore config if needed.
Version parity between HA peers is mandatory; mismatched versions break state sync and AP joins.
2. What monitoring and verification steps should be mandatory before closing the maintenance window?
Read the worked answer
Before closing:
- Confirm both controllers on the same firmware and HA sync healthy.
- Confirm all APs joined (count matches pre-upgrade baseline).
- Test authentication (staff 802.1X and a guest) end to end.
- Test client connectivity and roaming across the pair.
- Review controller logs for errors and confirm no alarms.
- Record results in the change ticket and confirm rollback artifacts are retained.
Only then declare the window complete; otherwise execute rollback within the window.