LTE Failover That Never Fails Over
A logistics company runs 25 depots on an SD-WAN overlay. Each depot has a primary fiber circuit and an LTE backup. The business requirement is sub-30-second failover for the dispatch application, which is latency-sensitive and must not drop sessions. The current policy is active/standby with a health-check probe to a public IP every 10 seconds, 3 failures to declare down. During a fiber cut at Depot 12, dispatch users lost connectivity for roughly 4 minutes before LTE took over, and existing sessions were dropped. Post-incident review shows the fiber was physically down for 4 minutes, the LTE link was up the whole time, and the probe target was reachable from LTE but the primary's probe still showed 'up' for the first ~3 minutes because the provider's upstream was still answering ICMP on a cached path.
1. Explain the gap between the 30-second requirement and the observed 4-minute outage, focusing on health-check design.
Read the worked answer
The failover timer is only as good as the health check's ability to detect the real failure. Here:
- Probe interval 10s × 3 failures = 30s minimum if the probe actually fails. But the provider answered ICMP for ~3 minutes on a cached/upstream path, so the primary looked 'up' while user traffic was black-holed.
- ICMP to a single public IP is a weak signal: it tests reachability of one target, not the actual forwarding path used by dispatch traffic, and it can be answered by an intermediate device.
- Active/standby with no pre-established LTE path means failover also requires overlay re-convergence, adding time after detection.
So the 4 minutes = ~3 minutes of false 'up' + detection + re-convergence. The requirement is violated by detection design, not by the LTE link.
2. Redesign the health check and failover policy to meet a 30-second, session-preserving requirement.
Read the worked answer
Make detection representative and make the backup path warm:
- Probe what matters: use an end-to-end probe that traverses the actual forwarding path to a target near the dispatch application (or multiple targets), not a single public ICMP responder. Prefer probes that require a real response from the destination service.
- Multiple diverse targets: require agreement across at least two independent destinations so a single cached responder cannot mask a cut.
- Tighter timers: probe interval 3-5s with 3 failures gives ~9-15s detection, leaving budget for re-convergence within 30s.
- Warm standby: keep the LTE overlay tunnel up and pre-established (active/active or hot standby) so failover is a path switch, not a tunnel build.
- Session preservation: enable flow-level failover / session continuity so established dispatch sessions are re-homed to LTE rather than reset. Confirm the application tolerates the brief path change.
- Asymmetric guard: ensure return path also uses LTE after failover.
Validate with a controlled fiber-pull test measuring detection time, switch time, and session survival against the 30s SLA.
3. The provider argues their ICMP response was correct behavior. How do you reconcile that with the design, and what operational safeguard do you add?
Read the worked answer
The provider is technically right: their upstream device answering ICMP is normal and not a fault. The design error is ours - we treated a single ICMP responder as proof the forwarding path was healthy.
Reconciliation:
- Accept that any single probe target can be answered by infrastructure that is not on the user-data path.
- Design detection around service reachability and path diversity, not a single responder's liveness.
Operational safeguard:
- Scheduled failover drills (e.g., quarterly per region) that pull the primary and measure detection/switch/session survival against the SLA.
- Probe-target diversity review as part of change management; never rely on one target.
- Alerting on probe anomalies (e.g., probe up but application latency rising) so false-'up' conditions are visible before an outage.
- Document the provider's behavior in the runbook so future engineers do not re-litigate it.