Meridian Financial: 74 Accounts, 11 Overlapping CIDRs, and a $30k Inspection VPC
Meridian Financial has 74 AWS accounts under AWS Organizations, organised into three OUs by environment, plus a network account and a log-archive account. The security mandate: every workload account ships VPC Flow Logs, CloudTrail, and GuardDuty findings centrally; no workload account may have an internet gateway; all egress must traverse a stateful, logged firewall; and no two VPCs may use overlapping CIDRs. Thirty platform teams deploy with Terraform and hold near-AdministratorAccess in their own accounts. Today each team builds its own VPC, NAT gateway, and VPC peering. Security found 11 overlapping CIDR pairs and 4 unintended peerings last quarter. Centralised egress will carry roughly 200 TB per month, and Finance is asking whether the inspection VPC is worth what it costs.
1. Design the preventive controls that stop overlapping CIDRs and unapproved connectivity at creation time, given that teams use Terraform rather than CloudFormation.
Read the worked answer
Why detective controls are insufficient here Config rules and Security Hub findings arrive after the resource exists and after the peering is live. The 11 overlaps exist because nothing prevented them. Prevention has to sit in the identity and allocation layers, because Terraform authenticates with an IAM principal in the workload account and does not pass through CloudFormation hooks.
CIDR: eliminate the choice Use VPC IP Address Manager. Create top-level pools carved from 10.0.0.0/8 per OU and per region, owned by the network account and shared to workload accounts with AWS Resource Access Manager. Teams no longer invent CIDRs; they pass an IPAM pool ID and receive an allocated /16 with /20 subnets. Overlap becomes structurally impossible, and allocations are visible centrally for future peering and Transit Gateway planning.
Connectivity: deny by default with SCPs Attach SCPs at the OU level that deny ec2:CreateVpc, ec2:CreateVpcPeeringConnection, ec2:CreateTransitGatewayVpcAttachment, ec2:CreateInternetGateway, and ec2:AttachInternetGateway, with a single named exception principal - the network platform role in the network account. Test condition keys for the IPAM pool requirement in your exact API version before relying on them; deny-listing the API outright is the version-independent control.
Defence in depth Enable VPC Block Public Access at the OU level so that even a successfully created internet gateway cannot route. Deploy AWS Control Tower proactive controls over the CloudFormation pathways so Service Catalog-provisioned accounts start compliant.
Segmentation One Transit Gateway with distinct route tables per OU, with associations and propagations that keep production, non-production, and shared services mutually unreachable except where explicitly routed. Only the egress route table has a default route into the inspection VPC.
Why this survives contact with 30 teams Every team keeps Terraform and near-admin in its own account. The constraint is that the account itself cannot obtain an overlapping CIDR or an unnotifiable path, which is exactly where the control belongs.
2. Finance asks whether the centralised inspection VPC is worth ~200 TB per month. Quantify it and recommend where to cut without weakening the security outcome.
Read the worked answer
The honest number Assume all 200 TB (204,800 GB) flows spoke, Transit Gateway, inspection VPC, Transit Gateway, egress VPC:
- Transit Gateway data processing is charged per GB on both attachments and each hop: 4 x 204,800 GB x $0.02 = about $16,400 per month.
- AWS Network Firewall processes 204,800 GB at $0.065 = about $13,300 per month, plus three firewall endpoints at $0.395/hour = about $870.
- All-in roughly $30,500 per month, or about $0.15 per GB, before any NAT or data-transfer charges.
The key question is not the price, it is the traffic mix Centralised inspection is being applied to 200 TB of traffic that is mostly not internet-bound. Typical enterprise VPC egress is dominated by S3, ECR image pulls, CloudWatch Logs, Systems Manager, and OS patch endpoints - traffic to AWS services that never needed a stateful firewall.
Where to cut
- S3 and DynamoDB gateway endpoints are free and remove that volume from the inspection path entirely. In most estates this is the largest single flow.
- Interface endpoints for ECR, CloudWatch Logs, STS, KMS, and Systems Manager at $0.01/GB remove the next tranche at a fraction of $0.15 per GB.
- Route on-premises-bound traffic over the Direct Connect path without hairpinning through the inspection VPC if that path is already inspected by the on-premises stack.
- After endpoints, the residual genuinely internet-bound flow is often 15-25% of the original. Run the firewall on 40 TB rather than 200 TB: about $6,100 per month all-in.
Recommendation Keep centralised inspection. It is the only place where FQDN-based egress rules, IDS/IPS signatures, and correlated flow logs exist - distributed NAT gateways at $0.045/GB are cheaper per GB but provide none of that, and would put the security mandate out of compliance in 74 accounts. Fund it by eliminating endpoint-able traffic, not by removing the control. Present the comparison to Finance per GB after endpoint optimisation, not before, because the before number is an artefact of the wrong traffic being inspected.
3. Specify the operating model that keeps 74 accounts compliant as the organisation grows, and name the two or three controls most likely to fail silently.
Read the worked answer
Account vending, not account building AWS Control Tower with Account Factory for Terraform. Every new account lands with the network baseline, logging role, backup plan, IAM permission sets, and tagging from Service Catalog - the same artefacts the network account defined. Enrolment is mandatory; an unenrolled account is the failure mode that defeats everything else.
Central logging that cannot be turned off Organisation trail to the log-archive account, with a bucket policy that denies deletion even to the management account, S3 Object Lock in compliance mode, and a cross-account KMS key the workload accounts cannot alter. Delegate GuardDuty, Security Hub, and Config administration to the security account so findings aggregate without per-account setup.
Identity over account-local IAM IAM Identity Center permission sets mapped to OUs, so tier of access is defined once. Terraform should assume a scoped CI role with OIDC federation rather than a long-lived administrator key - the near-AdministratorAccess the teams hold today is the reason the SCP layer must be airtight.
DNS must be solved or the IGW removal breaks things Share Route 53 Resolver rules and centralised endpoints via RAM. Removing internet gateways without central DNS resolution is the change most likely to produce a genuine outage that gets the network mandate rolled back.
Controls most likely to fail silently
- Tags for chargeback and ownership. Without a tag policy that denies resource creation on missing cost-allocation tags, coverage decays to 60% in two quarters and nobody notices until the showback report is meaningless.
- Account enrolment. A team creates an account outside Control Tower to bypass the SCP, and it is invisible until an audit. Alert on accounts not present in the organisation trail within minutes.
- Break-glass usage. A documented, MFA-gated, session-logged role with an alarm on every assumption. Without an alarm, break-glass becomes the default access path within a quarter.
Review rhythm Monthly: IPAM allocation report, SCP change log, unenrolled-account check, and cross-OU reachability diff against the intended topology. A topology nobody re-verifies is a topology that has drifted.