Firmware rollout to 42,000 field controllers with no remote rollback
A facilities-management vendor has 42,000 building controllers in the field running firmware v3.2, updated over cellular links; roughly 7,000 sites still have 2G-only connectivity. Firmware v4.0 introduces a new protocol that about 6,000 older gateway units cannot support, so those sites need a gateway swap before or alongside the controller upgrade. A latent defect in v3.2 makes scheduled heating fail after 400 days of continuous uptime, and about 9,000 devices are already past day 380. Remote rollback is not possible: a failed flash leaves the controller needing a site visit costing around £450 plus a three-day lead time. Standard customer contracts require 30 days' written notice before any firmware change. Engineering wants a single global push of v4.0 next month, arguing that the 400-day defect makes waiting more dangerous than upgrading.
1. How should the release manager package and sequence this work into release units?
Read the worked answer
A single global push is one release unit of 42,000 devices with no fast rollback - the worst possible fault-isolation profile. Break it up by capability and by risk.
Proposed units:
- v3.2.x patch - the minimum change that resets the 400-day timer without adding the new protocol. Small, quick, and applicable to every device including 2G and old-gateway sites. This addresses the urgent safety defect independently of the upgrade.
- v4.0 for gateway-compatible sites - staged: internal lab, then a pilot of 150-200 volunteer sites on good connectivity, then 5%, 25%, remainder, with defined halt criteria at each boundary (flash failure rate, post-upgrade fault tickets, connectivity loss rate).
- v4.0 plus gateway swap for the 6,000 incompatible sites - a separate release with an on-site work package, planned in regions so support capacity is not swamped.
Separating the defect fix from the protocol upgrade is the key judgement: it lets the safety issue be resolved in weeks without dragging 6,000 sites and a hardware programme into the same deployment.
2. How should the release manager handle the 9,000 devices past day 380 against the 30-day contractual notice?
Read the worked answer
Start notice immediately - day 380 to day 400 is a 20-day margin, so the contractual clock and the technical clock do not align. Waiting for a clean 30 days means some devices fail first.
Reasoning:
- Check whether the contract notice clause distinguishes fault remediation from feature change. A safety-related defect correction often falls under a maintenance or defect-remedy clause with shorter notice, but this must be read, not assumed.
- Issue notice for the v3.2.x patch now, describing it as a defect remediation, with a compressed but documented window and an explicit option for customers to defer in writing, recording the accepted risk.
- Prioritise by risk of the 400-day failure: devices closest to day 400, devices whose failure would affect occupied premises, and sites where a site visit is slowest (2G-only and remote sites need lead time, not the fast ones).
- Escalate to the emergency change route for any device that crosses day 400 before its scheduled update.
The release manager's job is to make the trade-off between contractual notice and physical failure visible to the service owner, not to silently pick one.
3. Remote rollback is impossible. How should that constraint change the release decisions and records?
Read the worked answer
When rollback is a £450 site visit with a three-day lead time, the release must be engineered so that rollback is rarely needed, and planned so that it is affordable when it is.
Decisions that follow:
- Lower risk appetite per batch. Smaller cohorts, longer dwell time before widening, and explicit halt criteria written before the rollout starts. A 2% flash-failure rate on 42,000 devices is 840 site visits; on a 200-device pilot it is four.
- Test the flash path itself, including interrupted downloads over 2G and power loss mid-flash. Most bricking happens in the update mechanism, not the firmware.
- Remote recovery mode where the hardware supports a fallback partition, and a documented depot-swap or advance-replacement option for units that fail.
- Budget the rollback. Reserve support capacity and cost for a defined worst case before starting; a plan that assumes zero failures is not a plan.
Records: each device's installed version, update timestamp, cohort and outcome should sit in the configuration management data, because with no remote rollback the version per device is the only way to know what is exposed after a partial rollout.