Sizing an HA Pair for a Bank DC Consolidation
A regional bank is consolidating two data centers into one. The core is a collapsed-core switch pair; two WAN routers run eBGP with two ISPs and ECMP. A new firewall pair will sit between the core and the WAN routers and will perform all inter-zone inspection with App-ID, threat prevention and TLS decryption. Measured peaks: 5.2 Gbps, 410,000 concurrent sessions and 95,000 new sessions per second at the open of business. The trading desk requires maintenance windows with no user-visible outage, and change control demands a tested rollback plan. Two engineers cover the estate, one on call at a time. Management wants both firewalls forwarding traffic "because we bought two", and wants ISP failover that does not blackhole established sessions.
1. Choose active/passive or active/active and defend the choice against the stated numbers and the "both firewalls forwarding" request. Name the evidence that would change your answer.
Read the worked answer
Size the chassis for the full peak first. 5.2 Gbps, 410k sessions and 95k CPS must fit on one chassis with the full feature stack (App-ID + threat + TLS decryption). Every HA pair loses one member in a failure, and an active/active pair that loses a member also halves capacity. So active/active buys you no failure-time headroom: you must buy the same single-chassis headroom either way.
What active/active actually costs here
- Requires HA3 interfaces and dedicated cabling; any packet that lands on the non-owning peer is forwarded across the HA3 link, adding latency and a new saturation failure mode.
- Two forwarding dataplanes and session-ownership concepts make packet captures and routing state harder to reason about for a two-person on-call rotation.
- The zero-downtime requirement is already met by active/passive graceful failover (
request high-availability state suspend), which drains and syncs sessions. "Both forwarding" is not needed to satisfy the trading desk.
Recommendation: active/passive, preemption disabled, with a documented suspend/failover runbook and an annual live failover test.
Evidence that would flip it: sustained load above roughly 60% of a single chassis's capacity with the real feature mix (decryption is usually the binding constraint) plus three years of growth, and proof that traffic is per-flow symmetric with HA3 headroom. Even then, verify datasheet numbers against decryption-enabled throughput rather than raw firewall throughput.
2. Describe the HA/interface and routing configuration that keeps eBGP sessions and established user sessions alive when one ISP fails. Where does source NAT fit in?
Read the worked answer
- Floating IPs on the untrust interfaces and on a loopback used for BGP peering. Peer with both ISPs from the floating address so the BGP session follows whichever chassis is active; otherwise failover presents a new peer address to the ISP and the session resets.
- Path monitoring on the untrust interface with several targets (ISP next-hop plus two public resolvers), failure condition
any. Link monitoring alone keeps the chassis active while the ISP is dead, which is the classic blackhole. - Route advertisement: advertise the aggregate only while healthy (conditional advertisement) or AS-path-prepend the backup ISP. Keep hold timers short enough for the business but long enough to avoid flapping.
- Source NAT to the floating (post-NAT) address, not to each ISP's own address. With a stable post-NAT source, the remote server keeps the same flow across failover; NAT-ing to the failed ISP's address guarantees resets.
- Session sync carries state to the standby, but sessions whose path was the failed ISP's own point-to-point endpoints (e.g. VPN peers terminating on that ISP's /32) will still die; plan for those separately.
- Avoid ECMP from the WAN routers to the pair. Give the routers a single next hop (the floating IP) so per-packet load balancing never splits a flow across the two chassis. Asymmetric return traffic is tolerable only while both directions traverse the same HA pair.
3. What must be done before cutover to make the rollback credible, given a two-person team?
Read the worked answer
- Build in parallel, then migrate one VLAN/zone pair per window instead of a big bang; keep the old path physically possible until the last window.
- Evidence pack: export device state, Panorama snapshot (if used), licence/registration confirmation, pinned content and software versions. Pin content before the window so behaviour doesn't change underneath you.
- Rehearse the failure paths in the window: fail the active member both ways, pull an ISP cable, and confirm path monitoring triggers failover and that the passive commits cleanly.
- Out-of-band management on a segment that does not depend on the new firewall or on the WAN routers being redirected.
- Rollback script with a time box: named WAN-router configs saved before change, documented procedure to restore the old next hop, a stated abort time, and someone empowered to call it.
- Success criteria and baseline captured beforehand (session count, decrypt resource usage, threat log rate, latency to the trading app) so the decision to keep or roll back is objective.