Intermittent Timeouts After the Batch Fleet Arrives
A three-tier VPC in us-east-1 hosts a payment service: public 10.0.0.0/24, app 10.0.10.0/24, data 10.0.20.0/24, no NAT gateway. App instances reach S3 through a gateway endpoint and Secrets Manager through an interface endpoint. A new batch fleet launched from a hardened AMI reports two different symptoms: S3 PUTs hang for roughly 60 seconds and then often succeed on retry, while Secrets Manager calls fail consistently. The app NACL allows inbound 443 from 10.0.0.0/16 and 32768-65535 from 0.0.0.0/0, and outbound 443 to 10.0.0.0/16 and 1024-65535 to 0.0.0.0/0. A recent change added a rule allowing 443 from the batch security group to the endpoint security group, but the endpoint group still only allows 443 from the original app group. tcpdump on a batch host shows SYN sent to the Secrets Manager ENI with no SYN-ACK returned.
1. Diagnose the two symptoms separately. What does the packet capture prove, and what does it rule out?
Read the worked answer
Secrets Manager: a security-group reference problem, proven by the capture. SYN with no SYN-ACK and no RST is a silent drop, which is what a stateful security group does when it has no matching inbound rule - it discards, it does not reject. A NACL would also drop, but the NACL allows 443 inbound from 10.0.0.0/16, and the endpoint ENI is in that range. So the NACL is exonerated and the endpoint security group is the suspect: it permits the app group, not the batch group. Adding a rule from the batch group to the endpoint group does nothing; security group references are evaluated on the receiving group. Fix the endpoint group.
S3: a routing problem, not a security-group problem. Gateway endpoints are not ENIs and involve no security group at all. Intermittent success with long hangs is the signature of a fleet spread across subnets where only some route tables carry the pl- prefix-list route to the gateway endpoint. Requests from a subnet whose route table lacks that association have no path (no NAT exists), the SDK retries with backoff, and a retry landing on a correctly-routed instance succeeds. Confirm by listing the route tables associated with every batch subnet and checking for the S3 prefix-list entry, and by checking whether failures correlate with AZ.
The general lesson: 'intermittent' plus 'eventually succeeds' points at path selection across subnets, while 'consistent failure with SYN and no reply' points at a stateful filter.
2. You also need the batch fleet to reach a partner's S3 bucket in eu-west-1 privately. What are the options and what is the catch with each?
Read the worked answer
A gateway endpoint is regional. It attaches to route tables in one region and provides a private path to S3 buckets in that region; it does not give you a private route to a bucket in eu-west-1. Do not promise it.
Options, with their real costs:
- Egress through NAT or an egress VPC in eu-west-1, reaching the partner bucket over the public S3 endpoint. Simple, but you lose the 'traffic never leaves AWS' property, and partner bucket policies must permit your account. Often acceptable because the payload is TLS to S3, but confirm the contractual language before assuming so.
- S3 Multi-Region Access Point with private connectivity. MRAP gives a single global name that routes to the bucket's region, but verify current support for reaching it through a VPC endpoint in your region before designing around it - this is precisely the detail that changes.
- Cross-region replication of the partner data into your region, then access it through your existing gateway endpoint. Cleanest private path, worst data-governance story, and it duplicates data the partner may not want copied.
Whatever you choose, keep it out of the batch subnet's default route: the request should be matched by an explicit prefix-list or endpoint route so that 'private' is a property of the route table, not of an application setting.
3. After the fix, how do you make the endpoint design least-privilege and stop this class of break recurring?
Read the worked answer
Three layers, in order of durability.
Endpoint policies. By default a gateway endpoint policy allows everything in the region with
Principal: *, so any principal in the VPC can reach any bucket. Restrict it to the specific buckets and the specific roles that need them. For the interface endpoint, restrict to the Secrets Manager ARNs in this account and region so a compromised host cannot enumerate another account's secrets through your ENI.Security-group hygiene for interface endpoints. Reference security groups, never CIDRs, and put the endpoint group under the same change control as the endpoint itself. A one-line Terraform review rule - 'endpoint SG changes require an app-owner approval' - prevents today's failure mode.
Detective controls that would have caught it in minutes. AWS Config rules on subnet route-table associations for each required prefix list, and a canary in each batch subnet that performs a signed S3 PUT and a Secrets Manager GetSecretValue once per minute. The canary catches asymmetric routing and SG drift that Config cannot see, and it fails from the same network position as the workload rather than from a monitoring host that was never the problem.
Finally, make the fleet's failure loud: a batch job that retries a 60-second timeout silently and succeeds is a job that hides a broken path for months. Alarm on retry counts, not only on failures.