I85PREP
⌘ K
FCSS Public Cloud Security study materials
FCSS Public Cloud Security study resources

FGCP pair across AZs behind a Gateway Load Balancer that never failed over

FCSS Public Cloud Security · Put your knowledge to work in context.

All case studies

FGCP pair across AZs behind a Gateway Load Balancer that never failed over

An e-commerce company runs a FortiGate-VM FGCP active-passive pair in AWS to inspect east-west traffic between 22 spoke VPCs attached to a Transit Gateway. Each FortiGate sits in a different AZ of us-east-1 inside a dedicated inspection VPC, behind a Gateway Load Balancer, with a GWLB endpoint in every spoke. The primary holds the Elastic IP and the secondary private IP used as the floating address. During a game day, engineers terminated the primary instance; traffic blackholed for 6 minutes until they manually changed a route, and CloudWatch showed the standby never became the active node. The SDN connector is configured with an IAM role. The team insists 'it worked in the lab', where both instances sat in the same subnet in the same AZ.

1. Diagnose the most likely reasons the standby did not promote and the outage lasted six minutes.

Read the worked answer

The lab/vs-production difference is the tell: same subnet and same AZ means Layer-2 FGCP worked by accident. Across AZs it cannot.

Why the standby did not promote

  • Heartbeat transport. FGCP hello/checksum traffic (UDP 703) and session synchronization (EtherType 0x8890/0x8891) do not cross subnets at Layer 2 and are not routed by default. Cross-AZ HA needs unicast heartbeat enabled (set unicast-hb enable) with a reachable hbdev, plus security groups that permit all traffic between the two members on the HA interfaces. If the SG only allows 703 inbound from the peer's primary IP, session sync silently fails and the pair degrades.
  • Eligibility. A standby with mismatched licensing, a lower HA priority than expected, override disabled, or a monitor statement on an interface that does not exist in that AZ will refuse to take over.

Why six minutes

  • The standby was probably never a healthy target in the GWLB target group. With GWLB in front, failover should be driven by the target group health check, not by FGCP moving an IP, so the outage length equals health-check interval × unhealthy threshold plus deregistration delay. Six minutes implies a slow probe (e.g., 30s × 3) or a target that was already unhealthy.
  • The EIP/private-IP move requires the SDN connector's IAM role to hold ec2:AssignPrivateIpAddresses, ec2:AssociateAddress, ec2:DescribeInstances, and route actions. A missing action leaves the floating IP stranded on a terminated instance.

Confirm by reading the FortiGate ha status history, the target group's health-check reason codes, and CloudTrail for failed AssociateAddress calls.

2. Redesign the deployment so a single-AZ failure recovers in under 60 seconds, and list the specific validation steps you would run before the next game day.

Read the worked answer

Target design

  • One FortiGate per AZ, always in the GWLB target group in both AZs (active-passive FGCP for session state, but both registered and probed), with GWLB cross-zone load balancing enabled so an AZ loss only removes that AZ's endpoint capacity.
  • Enable unicast HA over a dedicated HA interface, and put the HA members in a security group that permits all protocols between them. Do not reuse the data-plane security group.
  • Tune the target group: health check interval 10s, unhealthy threshold 2, healthy threshold 2, deregistration delay 30s (or 0 with connection draining handled by FortiGate), and point the probe at a port the FortiGate actually answers on the GENEVE-terminated interface - a probe to a closed port makes every target unhealthy and forces the load balancer to blackhole.
  • Scope the SDN connector IAM role to exactly the actions needed (ec2:DescribeInstances, ec2:AssignPrivateIpAddresses, ec2:UnassignPrivateIpAddresses, ec2:AssociateAddress, ec2:DisassociateAddress, ec2:ReplaceRoute) on the specific resources, and alert on AccessDenied in CloudTrail rather than discovering it during failover.

Validation checklist

  1. diagnose sys ha status and diagnose sys ha checksum show a synchronized pair with correct priorities.
  2. Terminate the primary from the console; confirm the target group marks it unhealthy within 20s, the standby becomes active, and the EIP follows.
  3. Run a continuous synthetic HTTPS transaction every 2 seconds from a spoke and record the longest gap - that is the real recovery time, not the console timer.
  4. Repeat for the GWLB endpoint AZ and for a single-spoke route blackhole, since endpoint failure is a different path than instance failure.