VPC Design for a Shared Services Hub
A company is consolidating four AWS accounts (prod, staging, dev, shared-services) into a hub-and-spoke network. The shared-services VPC hosts Active Directory domain controllers and a centralized egress NAT. Prod and staging must reach the internet only through shared-services egress, must reach on-premises over an existing Site-to-Site VPN that terminates in shared-services, and must NOT be able to reach each other. Dev needs internet egress but no on-premises access. Each VPC uses 10.0.x.0/24 ranges that currently overlap between prod and dev. The team proposes VPC peering between every pair of VPCs.
1. Evaluate the peering proposal and recommend an architecture that satisfies the isolation and routing requirements.
Read the worked answer
Peering fails on three counts here:
- Overlapping CIDRs. Prod and dev both use 10.0.x.0/24. Peering cannot be established between VPCs with overlapping ranges, and even where ranges don't overlap, peering is non-transitive - prod peered to shared-services does not give prod a path to on-prem through shared-services. You'd need a mesh plus manual route propagation, which doesn't scale and can't express "prod cannot reach staging."
- Non-transitivity kills the egress design. The requirement that prod/staging egress through shared-services is inherently a transit requirement. Peering can't do it.
- Isolation is hard to express. "Prod must not reach staging" is a policy you want enforced centrally, not by absence of peering links that someone can add later.
Recommendation: AWS Transit Gateway. Attach all four VPCs to a TGW in shared-services. Use separate TGW route tables: one for prod, one for staging, one for dev, each with routes to shared-services (for egress and on-prem) but no route between prod and staging. Isolation becomes a routing-table fact, not a hope.
- Overlapping CIDRs: resolve before attach. Re-IP dev (it's the least risky) to a non-overlapping range; TGW cannot route between overlapping attachments either.
- Egress: prod/staging/default routes point at shared-services NAT via the TGW attachment. Dev gets egress but its route table omits the on-prem route.
- On-prem: the VPN stays terminated in shared-services; TGW propagates those routes only to the route tables that should see them.
This centralizes policy, supports transit, and makes "who can talk to whom" auditable in one place.
2. After migration, prod instances can reach the internet but cannot reach on-premises hosts, while shared-services instances can reach on-premises fine. What do you check, in order?
Read the worked answer
Work outward from the working case (shared-services → on-prem works), which localizes the fault to the prod path.
- TGW route table for prod. Does it have a route for the on-prem CIDR pointing at the shared-services attachment? If the VPN routes propagated only to the shared-services route table and not to prod's, prod has no path. This is the most likely cause given the symptom.
- VPC route table in prod. Do the prod subnets have a route for the on-prem CIDR pointing at the TGW? A missing or wrong-target route here blackholes silently.
- TGW attachment and association. Confirm the prod VPC is associated with the prod TGW route table and that the shared-services attachment is propagating the on-prem routes. Association vs. propagation is a classic mix-up.
- Security groups / NACLs. If routing is correct, check that prod's egress and the on-prem hosts' ingress allow the traffic, and that no NACL blocks the return path.
- VPN tunnel and BGP. Confirm the tunnel is up and the on-prem prefixes are being advertised - but since shared-services works, the tunnel itself is likely healthy; the question is whether the prefixes reached prod's route table.
Order matters: routing before security, because a blackhole route produces a timeout indistinguishable from a security block, and you want to eliminate the cheaper explanation first.