Overlapping CIDRs Block a Shared-Services Network
A company has four AWS accounts connected by a Transit Gateway and a shared-services VPC. The production VPC uses 10.20.0.0/16; a staging VPC in another account also uses 10.20.0.0/16. On-premises uses 10.0.0.0/8 and reaches AWS over Direct Connect. A new payment service must call an internal API in the shared-services VPC and must also reach a third-party endpoint over the internet. Terraform manages the VPCs, but 60 security group rules were changed manually during an incident. One rule allows 0.0.0.0/0 to port 22 on a bastion, and a network ACL denies ephemeral ports for return traffic from the shared-services VPC. The team wants to connect the payment service without re-IP-ing production before the quarter ends.
1. Why will Transit Gateway attachment alone not solve the connectivity problem, and what are the realistic options?
Read the worked answer
Transit Gateway routes by destination prefix. If production and staging both use 10.20.0.0/16, the TGW cannot know which VPC should receive a packet for an address in that range; the routes conflict. The on-premises 10.0.0.0/8 summary makes this worse because it overlaps the production VPC range, so return traffic may prefer the on-premises route or be dropped as overlapping.
Realistic options: (1) use AWS PrivateLink to expose the specific internal API as an endpoint service, which avoids full VPC peering and CIDR overlap for that one service path; (2) re-IP staging first, since it is lower risk, then peer or route through TGW; (3) place a proxy/NAT instance in a non-overlapping range to reach the required API; or (4) use more specific on-premises routes if the actual on-premises subnets can be summarized more narrowly than 10.0.0.0/8. Re-IP-ing production is the most complete fix but does not meet the quarter-end constraint. PrivateLink plus a staged re-IP plan is the least risky path.
2. How should the team regain control of the 60 manually changed security group rules and detect future drift?
Read the worked answer
First, export the current rules and compare them with the Terraform state and last approved commit. Do not run a blanket apply that silently reverts incident fixes; classify each difference as legitimate emergency change, obsolete, or risky. Import legitimate rules into Terraform or encode them in a reviewed module, then remove the rest. For the 0.0.0.0/0 SSH rule, replace it with SSM Session Manager or a bastion restricted to known egress ranges, and remove public SSH after verification.
For drift detection, enable AWS Config rules for security groups and run terraform plan -detailed-exitcode on a schedule in each workspace. Send non-zero exit codes to the owning team. Enforce a policy that rejects world-open ingress on administrative ports and unexpected security group changes. Use separate Terraform workspaces or stacks for network security so application changes cannot mask rule drift.
3. The NACL denies ephemeral return traffic. What is the correct fix, and how would you verify it?
Read the worked answer
Network ACLs are stateless, so return traffic from the shared-services VPC to the payment service uses ephemeral destination ports. The NACL must allow inbound TCP ports 1024-65535 from the shared-services CIDR, and the corresponding outbound ephemeral range must also be allowed. If the client initiates to a service port such as 443, the request direction uses 443, but the response direction uses an ephemeral port. A deny on that range breaks the connection even when security groups allow it.
Verify from the payment service subnet using a controlled test: check the NACL association, confirm the route to the shared-services CIDR, inspect VPC Flow Logs for rejects on ephemeral ports, and test with curl or nc to the API port. Security groups are stateful, so do not add broad security group rules to compensate for a NACL problem. Fix the narrowest NACL range that covers the client ephemeral ports and keep the rule documented in Terraform.