Half the Requests Time Out - Flow Logs Point at the Wrong Layer
A retail platform runs a three-tier VPC in us-east-1. Public subnets (10.0.1.0/24, 10.0.2.0/24) hold an internet-facing ALB; private app subnets (10.0.11.0/24 in AZ-a, 10.0.12.0/24 in AZ-b) hold EC2 instances; RDS lives in isolated subnets. During a change window a new security group was attached to the app instances. Afterwards roughly half of all ALB requests time out and target health checks flap, but the instances show normal CPU, no application errors, and no reboots. Only the AZ-b app subnet's NACL was edited during the window; the AZ-a NACL is unchanged. The AZ-b NACL inbound allows TCP 443 from 10.0.0.0/16; its outbound allows TCP 443 to 10.0.0.0/16. The app security group allows 443 from the ALB security group and all outbound. VPC flow logs are on for every ENI.
1. A colleague says the new security group must be broken because the timing correlates. Using the symptoms and the flow logs, argue for or against that, and state what the flow logs will show.
Read the worked answer
Against. A security group is attached per instance, and the same group was applied across both AZs. A broken SG would fail on both AZs - you would see near-100% failure, not approximately 50%. The failure being cleanly partitioned by availability zone is the signature of a subnet-level, not instance-level control, and only the AZ-b NACL changed.
Flow log evidence to look for on instances in 10.0.12.0/24:
- Inbound records to port 443 with
ACCEPT- the SYN arrived and the NACL and SG both permitted it. - Corresponding outbound records with
REJECTfor the response, whose destination port is in the ephemeral range (1024–65535) and whose source port is 443.
That ACCEPT-in / REJECT-out pair on the same five-tuple is the fingerprint. It also explains the flapping health checks: the ALB opens a new connection for each check, and whichever ALB node and connection lands in a state that gets blocked times out, so the aggregate looks intermittent.
Mechanism: security groups are stateful - the response to a permitted inbound connection is automatically allowed and never evaluated. NACLs are stateless - the return packet is a separate flow and must be explicitly permitted. Because the client's source port is ephemeral, the NACL must permit outbound TCP 1024–65535, not 443.
2. State the exact rule change, then explain why the tempting fix - allow all traffic in both directions on that NACL - is the wrong call here even though it works.
Read the worked answer
Correct change: on the AZ-b app-subnet NACL, add/replace the outbound rule with allow TCP 1024–65535 to 10.0.0.0/16 (or, tighter, to the two ALB subnet CIDRs 10.0.1.0/24 and 10.0.2.0/24). Keep inbound 443 from the ALB subnets. Remember rule-number ordering: a lower-numbered deny still wins, so place the allow below any deny you intend to keep.
Why not allow-all:
- The NACL is your only control that survives a security-group change. App-tier instances have broad outbound in their SG; a permissive NACL removes the last subnet-level barrier to egress, which is exactly the exfiltration path abused in most EC2 compromises.
- A
0.0.0.0/0allow on both directions makes the two app subnets effectively flat, so the RDS isolated subnets become the only thing between the app tier and the database. - It converts a 20-minute targeted fix into an undocumented permanent expansion of the trust boundary, and nothing will alert on it later.
Compensating practice: treat NACLs as coarse, intentional, subnet-scoped guardrails, not as fine-grained filtering - keep them few and documented. Add an AWS Config custom rule or a config-rule lambda that flags NACL entries with 0.0.0.0/0 on the app subnet, and check for ephemeral-range coverage after any NACL change. That turns this class of outage and this class of shortcut into an automatic finding.
3. Separately, the app tier must move 8 TB/day of build artifacts to S3 without a NAT gateway. Design the connectivity and the control that keeps it from becoming a general-purpose internet path - and note the one place the NACL still matters.
Read the worked answer
Connectivity: an S3 gateway VPC endpoint attached to the app subnets' route tables. Gateway endpoints are free, keep traffic on the AWS network, and require no NAT gateway, no public IP, and no internet gateway path. Add the prefix list to the route tables and you are done.
The control is the endpoint policy, not the security group. Gateway endpoints have no security group and no ENI - the only knob is the endpoint policy. Write it to allow exactly what the app needs:
s3:PutObjectonarn:aws:s3:::corp-artifacts/builds/*from the app role ARN,s3:GetObjecton the same prefix,s3:ListBucketscoped with as3:prefixcondition,- deny everything else, and deny any request where
aws:PrincipalArnis not the app role.
That policy is what prevents the endpoint from becoming an unrestricted S3 path for anything else running in those subnets. Without it, the default policy ("Action": "s3:*", "Resource": "*") is a data-exfiltration channel to any bucket the role can reach.
Where the NACL still matters: with a gateway endpoint, traffic is routed via the prefix list through the same subnet, so it is still subject to the subnet NACL. If the artifact traffic uses TCP 443 outbound to the S3 prefix list, the ephemeral-return rule you just fixed also governs this path - a future 'tightening' of the AZ-b NACL would break artifact uploads with the same confusing symptoms.