OTA Campaign for 20,000 Smart Meters
A utility operates 20,000 4G-connected smart electricity meters on platform firmware 1.2.3. A security patch in 1.3.0 must reach all devices within 30 days. The image is 3.2 MB; each meter has 4 MB flash with one application slot plus a small recovery bootloader, and meters are mains powered. The last campaign pushed to 5% of the fleet at once during business hours: the local 4G cell uplink saturated, 1,400 meters timed out mid-download, and 600 rebooted into the bootloader and required truck rolls at $80 each. The platform supports device shadow, batch tasks, rate limiting and signature verification, but the meters do not have A/B slots. Field engineers can also visit with a laptop and optical port.
1. Redesign the campaign: scope, timing, concurrency and safeguards.
Read the worked answer
The previous failure was a capacity and sequencing failure, not a firmware failure. Fix the plan:
- Canary: 20-50 meters in one cell, watched for 48 hours on version, reboots and error codes.
- Rate: 20,000 devices over 30 days is about 670 per day. Limit concurrent downloads per cell to, say, 20 and globally to a few hundred, so the 4G uplink is not saturated.
- Timing: window 01:00-05:00 local, outside peak load and outside grid switching work.
- Pre-checks before download: mains power present, RSRP above threshold, free flash confirmed, current version 1.2.3, and a valid signature on the package.
- Download to a staging area fully, verify hash and signature, then flash. Do not stream-flash on a link that can time out.
- Retries: three attempts with exponential backoff, then park the device for manual review; halt the batch automatically if the failure rate exceeds 2%.
- Off-peak retries for parked devices, and a documented rollback to 1.2.3 if a systematic fault appears.
The canary and the per-cell rate limit are the two controls that would have prevented the 1,400 timeouts.
2. How do you recover the 600 meters stuck in the bootloader, and how do you prevent a repeat without A/B slots?
Read the worked answer
Recovery first. The shadow and task history identify which devices stopped at which step, so triage into three groups:
- Devices whose bootloader has a network stack: push a minimal recovery image over the air in small chunks with long timeouts and no concurrency limit conflict, during the off-peak window. Verify the signature before the bootloader writes it.
- Devices with no network stack in the bootloader: they need a visit, so batch the truck rolls geographically to cut cost, and use the optical port with the same signed recovery image.
- Devices that reappear with 1.2.3 but a corrupted flag: re-run the normal campaign for them after the fleet campaign.
Prevention without A/B slots:
- Keep a protected golden image in flash that the bootloader can fall back to.
- Never flash directly from the network; download, verify hash and signature, then flash while the bootloader supervises with a watchdog and brown-out detection.
- Treat power loss and timeout as expected events, not exceptions, and log the exact step reached so recovery is targeted rather than a truck roll.
3. Which platform features and KPIs would you use to run and prove this campaign?
Read the worked answer
Use the platform rather than spreadsheets:
- Device shadow for a single source of truth per meter: current version, target version, last campaign ID, last error code, signal quality, power status.
- Batch tasks with a filter that selects by version and cell, plus rate limiting and a scheduled start so the window is enforced by the platform.
- Signature verification and package inventory so only approved images can be dispatched, and every dispatch is attributable to an operator and a change record.
- Reporting on shadow-reported state rather than on task submission, because task submission does not mean the device flashed.
KPIs to publish daily: percentage of fleet on 1.3.0 against the 30-day target, download success rate, flash success rate, retry count per cell, median download duration, devices in bootloader, cell-level concurrency and failure codes ranked by frequency.
An automatic pause rule, for example a stop if any cell shows more than 2% failures in an hour, turns the platform from a broadcast tool into a controlled release system and protects the remaining fleet.