A shared mesh key across 60,000 meters and a rollout deadline
An electricity distributor operates 60,000 smart meters with a 15-year service life. Of these, 8,000 are an older model whose radio module cannot accept over-the-air firmware images larger than 32 KB, and the vendor has announced end of support in 18 months. Meters sit in customers' premises, communicate over a private mesh to 400 concentrators, and use a shared symmetric key injected at manufacture for mesh authentication. Field technicians provision meters with a handheld tool protected by a single site-wide password. A penetration test found that anyone with physical access to a meter can extract the shared key and impersonate any meter on the mesh. The distributor must also commission 20,000 new meters next year under a regulatory rollout timetable, and the regulator has asked for the remediation plan.
1. What can be fixed now, what must be compensated for, and how do you decide?
Read the worked answer
Separate remediable design flaws from structural ones.
Fixable in the field or back end: the provisioning tool - replace the site-wide password with per-technician accounts, MFA at the back end, time-bound tokens and full session audit; concentrator backhaul - mutual authentication, segmentation so the meter mesh cannot reach corporate or billing systems; monitoring for duplicate identities, replay and abnormal command rates; tamper evidence and seal inspection; key rotation on models where the OTA budget allows it.
Not fixable at scale: the shared key in 8,000 unsupported meters. Replacing it requires visiting every premises and the radio still cannot hold a larger secure image. This becomes a dated replacement commitment with compensating controls in the interim.
Decide by attack path. The scenario is impersonation leading to false consumption data, disconnect or reconnect commands, load manipulation and privacy loss. Compensating controls must target that path: physical access (tamper detection, sampling inspections), mesh behaviour (baselines, command rate limits, dual authorisation for high-impact commands), and containment (segmentation, no direct path to billing).
Document it. A treatment plan with residual risk owned by the accountable executive, an 18-month replacement schedule aligned to vendor end-of-support, and board visibility. The end-of-support notice is contractual leverage - use it in the replacement negotiation rather than absorbing the cost silently.
2. What requirements do you impose on the 20,000 new meters so the same problem is not repeated?
Read the worked answer
Treat procurement as a security control. The failures here - shared key, no crypto-agility, small update budget, weak provisioning - were all decided before deployment.
Requirements:
- Unique per-device credentials or keys derived from a root held in a hardware security module or secure element; no shared mesh-wide secret.
- Secure boot and signed firmware, with an OTA image budget sized for the full 15-year life, including future cryptographic algorithms (crypto-agility).
- A patch and vulnerability-handling commitment with response times, a disclosure channel, and a stated end-of-support date with successor commitments.
- Software bill of materials and component transparency.
- Secure manufacturing and key-injection process, with audited handling and no keys recoverable from the field device.
- Staged rollout and rollback support, plus telemetry sufficient for central monitoring.
- Audit rights and defined remedies if requirements are not met.
Verify rather than accept. Include a penetration test and a key-extraction attempt in acceptance testing, on sample units, before volume delivery. Make acceptance conditional on results.
Escalate the trade-off. If a supplier cannot meet a requirement, that is a procurement risk decision to be recorded and owned at executive level, not quietly waived by the project team under timetable pressure.
3. How do you run the firmware and key changes safely at this scale, and how do you handle an incident mid-rollout?
Read the worked answer
Stage by cohort. Pilot on a small, low-consequence group with a rollback path proven in advance; measure failure and rollback rates before widening. Sequence by model capability and by criticality of the premises supplied. Publish maintenance windows and customer communications, because meter work can interrupt supply and billing reads.
Control the change. Freeze the configuration baseline during rollout, log every command, and require dual authorisation for bulk operations. Use rate limits so a faulty push cannot take out a whole cohort at once. Keep a known-good image per model, and define abort criteria before starting rather than during.
Watch during and after. Monitor for meters dropping off the mesh, authentication failures, duplicate identities and command error rates. A spike in any of these is a stop signal.
Incident mid-rollout. Separate the two problems: a rollout fault is handled by rollback; a security incident is handled by the incident plan. For a suspected key compromise, the immediate actions are to limit blast radius (segment concentrators, restrict high-impact commands, raise monitoring), preserve evidence, and assess notification duties to the regulator, which has already asked for the plan. Do not delay containment to protect the timetable.
Feed back. Every failure in the rollout should update the treatment plan, the supplier requirements and the test criteria, closing the improvement loop rather than being filed as a project issue.