Dual-Data-Center Active/Active HA with Dual-ISP BGP
An enterprise with two data centers (DC-A and DC-B) and 12 branch offices is replacing legacy firewalls. Each data center has two 10 Gbps ISP links from different providers; branches connect through MPLS with an internet backup. The security team requires sub-second stateful failover, symmetric return paths for SIP and FTP, and no NAT pool exhaustion after a single link loss. The proposed design uses an active/active PA-5450 pair in each data center, BGP-advertised floating IPs, and ECMP across both ISPs. Current peak is 7.5 Gbps per data center, 480,000 concurrent sessions, and 95,000 new sessions per second. The network team plans iBGP between the firewalls and the core, eBGP to each ISP with identical MED values, and static routes from branches. The business wants dynamic branch failover between MPLS and internet without changing branch routers.
1. Evaluate the active/active HA and ECMP proposal. What design changes, if any, should be required before approval?
Read the worked answer
Active/active HA is supported, but the design must account for session ownership and return-path symmetry. With ECMP across two ISPs and identical MED values, inbound traffic can arrive on either firewall or either ISP, while SIP and FTP ALGs and source NAT may pin return traffic to a different path. That creates asymmetric routing, out-of-window TCP drops, and failed ALG transformations.
I would require one of two models: (1) active/passive per data center, with the active node owning floating IPs and advertising the ISP summary, or (2) active/active only if ISP links terminate on the same pair and ECMP is strictly per-flow with session synchronization verified for both directions. For either model, use BGP conditional advertisement or AS-path prepending so only the preferred ISP is advertised inbound, and do not rely on identical MED. Also validate 95,000 new sessions per second and 480,000 concurrent sessions with threat inspection and decryption enabled, not just raw throughput. The PA-5450 may be sufficient, but the tested CPS and session limits at the intended feature profile must be confirmed.
2. How should ISP redundancy and branch failover be implemented to preserve symmetry and sub-second recovery?
Read the worked answer
For ISP redundancy, I would use a floating IP owned by the active firewall, advertise a more specific summary only from that node, and run BFD to each ISP next hop for sub-second detection. Outbound traffic should use policy-based forwarding with health probes or BGP local preference, not equal-cost ECMP, unless the design has been proven symmetric for stateful sessions. NAT pools should be sized for full load on one ISP, not split across both.
The branch requirement is the conflict: branch routers use static routes, so they cannot dynamically select MPLS or internet without a routing change. Options are to add a tunnel overlay with dynamic routing, update branch routers to run BGP or IPsec backup, or centralize branch internet breakout at the data center. I would not approve a claim of dynamic branch failover while branch routers remain static. Recovery should be tested with SIP and FTP sessions crossing the failure, because those protocols expose asymmetry and ALG state problems that ping tests will miss.
3. What capacity and high-availability validation would you require before cutover?
Read the worked answer
I would require a test plan that covers four areas. First, failover under load: generate 7.5 Gbps with 480,000 concurrent sessions and 95,000 CPS, then fail the active node and one ISP link while SIP, FTP, and HTTPS transactions are in flight. Second, routing convergence: measure BFD and BGP withdrawal times, verify floating IP movement, and confirm that the standby does not advertise the ISP prefix prematurely. Third, NAT and ALG validation: exhaust one NAT pool deliberately and confirm that sessions move to the surviving pool without dropping established flows. Fourth, feature-path performance: repeat the load with threat prevention, URL filtering, and decryption enabled, and confirm dataplane CPU and session table headroom. I would also verify HA3 synchronization for every virtual router and logical interface used by the branches, because a partially synchronized standby can pass basic failover tests yet drop MPLS traffic.