AquaSense: Maintaining a Soil-Moisture Fleet at 18 Months
AquaSense operates 4,800 solar-powered soil-moisture sensors across commercial farms, reporting hourly over cellular. At 18 months the fleet shows: 212 devices offline, with 60% of those at sites more than 40 km from the nearest town; battery voltage trending below the model, with extrapolated life of 3.2 years against a design target of 7; a carrier notice that 3G service in the region ends in 14 months; a device root certificate expiring in 9 months with no rotation tooling; and a 1.4 firmware release that failed on 3% of devices, of which two-thirds never recovered remotely. Support tickets peak after rain events. The operations team of four handles roughly 40 truck rolls per month and has no remote diagnostics beyond last-seen timestamps and battery voltage.
1. Triage the four issues into a prioritised programme and justify the ordering with risk, not ease.
Read the worked answer
Ordering by consequence-if-unaddressed times likelihood, not by how simple the fix is.
1. Certificate expiry (highest). Nine months away, and the failure mode is fleet-wide simultaneous: every device that cannot complete a TLS handshake stops reporting on the same day. A 212-device outage is manageable; a 4,800-device outage ends the product. The absence of rotation tooling is the real finding, because a certificate that must be replaced without tooling becomes a truck roll per device.
2. 3G sunset (second). The deadline is hard, public and unavoidable. Fourteen months sounds generous until you count procurement, certification, field swaps and a winter window. It also has a hidden component: '3G' in the notice may include modules that are 3G-only with no LTE fallback, so the affected population must be identified from the device inventory, not assumed to be all 4,800.
3. Battery and the 212 offline (third, and they are the same investigation). The 60% concentration of offline devices far from town is the tell: weak signal means higher spreading factors, more retransmissions and more join attempts. That is an energy problem and a coverage problem at once, and it explains why extrapolated life fell from 7 to 3.2 years.
4. Firmware failure rate (fourth, but blocks the others). You cannot solve 1, 2 or 3 without reliable remote update, and a 3% failure rate with two-thirds unrecoverable is disqualifying. It becomes priority one the moment it is framed as an enabler.
Sequencing: fix update reliability, then certificate rotation using it, then the module migration, with the battery/coverage work running in parallel as diagnosis.
2. The battery life is 3.2 years instead of 7. Describe how you would diagnose the root cause and what evidence you need from the fleet.
Read the worked answer
A 2.2x shortfall is too large for a single cause; assume two or three compounding factors and instrument for them.
Build the energy budget from measurements. For each device, the dominant terms are radio transmit energy, receive windows, join attempts, sensor warm-up and sleep current. Transmit and receive time scale directly with spreading factor, so the first question is: what SF is the fleet actually using, and how often does it retransmit? A field reporting hourly at SF7 uses a fraction of the energy of one at SF10 with 15% retransmission.
Evidence to collect. Historical transmit counts and acknowledged uplinks per device; SF distribution over time; join request frequency (repeated joins mean poor coverage or a provisioning fault that has never really resolved); reset and brownout counters; measured sleep current on a sample of returned units; temperature history, since lithium cells lose substantial capacity below freezing and self-heating is not available in this form factor.
The rain-event signal. Tickets peaking after rain suggests water ingress into the enclosure or connector, causing leakage current or corrosion-driven resets, or foliage and soil moisture attenuating the link. Any of these shows as a cluster of resets and retransmissions at the same timestamps as the ticket spike. Correlate the two data sets; if they align, the fault is hardware protection, not firmware.
Firmware audit. Check whether any release leaves the receiver enabled longer than the design assumed or fails to disable a peripheral before sleep. Review the 1.4 release notes against the power state machine; a 3% failure rate hints at other latent defects.
Validation. Rebuild the model with measured, not datasheet, parameters and predict life for a weak-signal site; if the model then matches 3.2 years, the design target was never achievable at those sites and the fix is coverage plus reporting policy, not a better battery.
3. Write the migration plan for the 3G sunset and certificate rotation, including the safeguards that make it survivable.
Read the worked answer
Treat both as a single programme, because the certificate work is the dress rehearsal for the module migration.
Inventory first (weeks 1-2). Extract module model, firmware, SIM profile and certificate identity per device. The affected population for the sunset is devices whose modules cannot attach to LTE; it is probably a subset, and swapping the wrong population wastes the window.
Fix update reliability before touching the fleet (weeks 2-6). Root-cause the 1.4 failures: 3% failing with two-thirds unrecoverable points to a missing rollback path, no health check after boot, or power loss during flash write on solar-powered devices. Require A/B slots with automatic rollback, resume with block checksums, updates inhibited below a battery threshold, and a boot counter. Prove it on a bench rig with forced brownouts before any production update.
Certificate rotation with overlap (weeks 6-10). Load both the old and new root into the trust store during a transition window, so devices rotate without a flag day. Roll out new credentials to 1% of the fleet, watch success rate, then 10%, then the rest. Keep the old root valid for at least 60 days after the last device is confirmed rotated. Alarm on handshake failures per hour so a problem is visible in minutes, not at expiry.
Module migration (months 4-14). Dual-mode modules where possible; swap 3G-only devices in regional batches, prioritising the remote sites that already have weak coverage and will benefit most. Schedule around harvest and winter access.
Safeguards throughout. Staged canary rollouts, a documented rollback at every stage, spares held for the failure rate, remote diagnostics (SF, retransmission rate, reset counters, last handshake) so the team of four is not driving to find out what happened, and a weekly dashboard tracking percentage migrated, percentage rotated, and unresolved failures against the deadline.