TransCargo: Meeting 99.95% Site Availability Across 72 Distribution Centers
TransCargo runs 72 distribution centers. Each has an SRX345 with a fiber primary and LTE secondary, an EX2300 stack, and Mist APs. The SLA is 99.95% per site, but last quarter 9 sites exceeded 40 minutes of downtime. BGP failover takes 90–120 seconds because hold timers are default and the LTE link is a floating static route that depends on an IP SLA. LTE overage charges hit $38,000 in one month when a fiber cut went unnoticed for 11 hours. Monitoring is SNMP polling every 5 minutes into an aging syslog server; there is no configuration backup, and firmware spans 14 Junos versions. The new COO wants a reliability program before peak season, not another dashboard. Capital is limited to $600K, and two sites have no second carrier.
1. Redesign the WAN failover and degraded-mode behavior to meet the SLA.
Read the worked answer
The failover time is a protocol design problem, not a carrier problem. Replace the floating static with BGP on both transports, or keep static but add BFD with sub-second timers and an IP SLA tracker tied to a tracked object. Test that LTE carries traffic only after fiber loss and returns to standby, with a preemption delay to prevent flapping.
For the two single-carrier sites, design local survivability: branch SRX with local internet breakout for cloud applications and a documented degraded mode. Consider Juniper SSR if application steering is required, but do not add it only for failover. Set LTE usage alarms at 70% of plan and alert when failover lasts longer than 10 minutes. The 99.95% target must be measured per site, so instrument the WAN edge and report monthly. The metric is time-to-restore, not whether a backup link exists on paper.
2. What monitoring and observability would you add, and what would you deliberately not add?
Read the worked answer
SNMP every five minutes is availability monitoring, not observability. Add streaming telemetry or Mist WAN Assurance for per-link latency, jitter, loss, and application experience; keep SNMP for legacy devices. Define SLIs: site availability, failover duration, LTE minutes, and top application latency. Alert on symptoms users feel, not just interface state.
Centralize syslog with retention that satisfies audit, and correlate SRX logs with path telemetry. Use Marvis or equivalent for anomaly detection only after the data is clean. Publish a weekly reliability scorecard per site so the COO sees trend, not a dashboard. Before peak season, run failover drills at three representative sites and record actual times. Deliberately do not add another dashboard; add data quality, alert ownership, and drill evidence.
3. How do you fix configuration drift and firmware sprawl with a limited budget?
Read the worked answer
Configuration drift is a reliability defect. Build a golden configuration per site profile, store it in a version-controlled repository, and back up running configurations automatically after every change. Use Junos PyEZ or a network management platform to compare against golden, and report drift weekly.
Standardize firmware into two supported releases: one for distribution centers with LTE, one for simple sites. Schedule upgrades by site cohort with a rollback image pre-staged. Restrict change access to a small team and require a change ticket that references the configuration diff. The $600K should fund BFD-capable circuits where possible, LTE plan changes, and tooling, not another dashboard. Track mean time to repair and repeat-incident rate as the outcome.