The NAT Gateway That Ate the Budget
A data-processing workload runs on 40 EC2 instances in private subnets across three AZs. Each instance pulls a 2 GB container image from Amazon ECR every hour and writes results to S3. The team uses a single NAT Gateway in one AZ. Monthly data processing charges are $9,400, of which $6,100 is NAT Gateway data processing. The instances also show elevated latency during peak processing. A cost review flags the NAT spend as the top target.
1. Explain where the NAT data processing charge comes from and why the single-AZ design also creates a reliability problem.
Read the worked answer
Cost source: NAT Gateway charges $0.045/GB processed plus hourly. Every byte the 40 instances pull from ECR and push to S3 traverses the NAT Gateway. At 2 GB/hour × 40 instances × 720 hours ≈ 57,600 GB/month, that is roughly $2,600 just for ECR pulls, plus S3 traffic - consistent with the $6,100 figure. The traffic is unnecessary because both ECR and S3 have VPC endpoints.
Reliability problem: A single NAT Gateway in one AZ means (a) an AZ failure takes down egress for all three AZs, and (b) cross-AZ traffic from instances in the other two AZs incurs inter-AZ data transfer charges and added latency. The elevated latency during peak is consistent with all egress funneling through one AZ.
So the design is both more expensive and less resilient than the alternative.
2. Design the remediation that cuts cost and improves reliability, and estimate the impact.
Read the worked answer
Remediation:
- Add a Gateway VPC endpoint for S3 - free, removes all S3 traffic from NAT. S3 traffic is the largest single component.
- Add an Interface VPC endpoint for ECR (
ecr.apiandecr.dkr) plus S3 gateway endpoint for image layers - removes ECR pull traffic from NAT. Interface endpoints have an hourly + per-GB cost, but far below NAT's $0.045/GB at this volume. - Deploy one NAT Gateway per AZ (three total) so each AZ's private subnet routes through its own NAT. This removes the cross-AZ dependency and the single point of failure, at the cost of two extra hourly NAT charges (~$65/month) - trivial against the data processing savings.
- Cache images locally - pull once per host and share, or use a local registry mirror, to cut the 2 GB/hour repetition.
Estimated impact: S3 + ECR traffic is likely 80-90% of the NAT data processing. Removing it could cut the $6,100 to well under $1,000, while the per-AZ NATs add ~$65/month. Net savings on the order of $5,000/month, with improved latency and AZ-failure resilience.