North-South Throughput Does Not Scale on a Single Stateful Tier-1

The bottleneck is the Tier-1 service router, not the edge cluster.

A Tier-1 with stateful services (NAT, gateway firewall) runs its service router in active-standby: exactly one edge node hosts the active SR for that Tier-1 at any moment. Every north-south flow for all 900 workloads is therefore processed by one edge node, regardless of how many edges the cluster contains. Adding edges adds failure domain and aggregate capacity for other service routers - not for this one.

Why ECMP did not help. ECMP on the Tier-0 uplinks distributes external traffic across the Tier-0 service routers, which are active-active. But the Tier-0 SR on an edge that does not host the Tier-1 SR has nowhere to send the packet except across the /31 transit segment to the edge that does. So traffic still converges on the single active T1 SR, and the extra hop consumes edge capacity without adding throughput.

Why one node looks idle. That node holds the standby SR for the Tier-1. It receives little or no north-south traffic in steady state - it only forwards the hairpinned traffic described above. Near-zero throughput plus uneven load across the others is the signature of one active SR on one edge plus transit hairpinning, not of a broken ECMP hash.

East-west is unaffected because the distributed router lives on the hosts and never touches an edge.

Option A - partition into multiple Tier-1s (recommended). Split tenants or application families into several Tier-1 gateways, each with its own SR. Each SR has its own active edge, so with four edges you can spread four stateful T1s and get real north-south scale, plus better failure isolation. Cost: a broader routing and policy surface, and a migration of NAT/firewall rules from the single T1 into per-tenant policies. This is the only option that uses the hardware already purchased.

Option B - scale up the edge form factor. Since one node does the work, give that node more capacity: larger VM form factor, or bare-metal edges. Simple, no architecture change, but it is a single-node ceiling and a bigger blast radius on failure.

Option C - move the stateful services to the Tier-0. The Tier-0 supports NAT and gateway firewall, which removes the single T1 SR bottleneck in some designs. Validate it against the supported design guide first - stateful services on an active-active Tier-0 have their own constraints and are not a drop-in equivalent.

Not an option: enabling more ECMP paths or adding a fifth edge. Both were tried in spirit already and cannot fix a single-active service router.

I would fund Option A and keep Option B as a stopgap for the next peak.

Confirm the active SR location first.

• Identify which edge node currently hosts the active SR for the Tier-1, and confirm the other edges host only standby SRs. If the active SR moved during the test window, throughput samples are not comparable.
• Correlate edge-node throughput graphs with the active/standby state over the same interval. The active node should carry the vast majority of north-south bytes; a node that only carries hairpinned transit traffic will not.

Confirm the hairpin.

• Look at traffic volumes on the /31 transit segment. Sustained inter-edge transit traffic during north-south bursts is direct evidence that external traffic is entering on a non-owning edge and being forwarded to the active SR.

Size the fix.

• Per-Tier-1 peak north-south throughput and flow count, not just aggregate site throughput. This is the number that determines how many T1s you need and how they should be partitioned.
• Flow count and connection rate matter as much as bits, because stateful NAT and firewall scale on sessions.
• Headroom target should be based on peak plus the loss of one edge - with stateful T1s, the surviving nodes must absorb the failed node's active SRs.

Only then pick the partition boundary: splitting by tenant is easy to explain but may not balance load; splitting by measured flow volume usually balances better.