502s and Payment Timeouts After Security Group Hardening
FinLedger has a VPC 10.30.0.0/16 with three subnets: public 10.30.1.0/24 for an Application Load Balancer, private app 10.30.2.0/24 with four app servers, and private DB 10.30.3.0/24 with RDS. Security groups were tightened last week. sg-lb allows inbound 443 from 0.0.0.0/0 and all outbound. sg-app allows inbound 8080 from sg-lb, outbound 3306 to sg-db, and no other egress. sg-db allows 3306 from sg-app. The app health endpoint returns 200 when curled locally on port 8080. Since the change, the ALB marks about half of targets unhealthy and returns 502. The app also cannot reach the payment API at api.payment.example, which resolves to public IPs. App servers have no public IPs. The team wants to keep app and DB private and avoid 0.0.0.0/0 egress. A NAT gateway exists in the public subnet, but the private app route table has no default route.
1. Explain the 502s. Why does allowing 8080 from sg-lb not reliably admit the ALB health checks, and what rule would you add?
Read the worked answer
The ALB health check does not originate from an instance that is a member of sg-lb. Security-group referencing matches traffic from elastic network interfaces that carry the referenced group; a managed load balancer's health-check traffic is not tagged with sg-lb. The app servers accept direct local curls, which is why the health endpoint itself looks healthy. Add an inbound rule on sg-app for TCP 8080 from the load balancer's subnet CIDR, 10.30.1.0/24, or from the documented load balancer health-check IP range. Keep the rule as narrow as the service documentation allows. Then confirm target health and watch for 502s disappearing. Do not open 8080 to 0.0.0.0/0; that exposes the app tier directly.
2. External payment calls time out from the app subnet. Describe the route and security-group changes, and a tighter alternative to 0.0.0.0/0 egress.
Read the worked answer
Add a default route 0.0.0.0/0 to the NAT gateway in the private app route table. Add an outbound rule on sg-app for TCP 443 to the payment provider's published CIDR ranges, not to all destinations. If the provider's ranges are unstable, deploy a small egress proxy in the public subnet with a static address, allow only the proxy from sg-app, and let the proxy enforce the allowlist. Keep sg-db unchanged; the app still reaches the database only on 3306. This preserves private app servers while giving them controlled outbound HTTPS. Validate with a curl from an app instance and check NAT gateway metrics for drops.
3. How would you verify both fixes without weakening the DB tier?
Read the worked answer
Verify in layers. First, check ALB target health and confirm 8080 is reachable from the load balancer path, not just localhost. Second, run an HTTPS request from an app server to the payment hostname and confirm DNS resolution, route selection, NAT translation, and the egress rule. Third, test that the DB still accepts 3306 only from sg-app and rejects a connection from the public subnet. Capture evidence in flow logs or temporary packet captures, then remove broad test rules. Do not add any 0.0.0.0/0 rule to sg-db or sg-app during troubleshooting; use source-specific rules and time-bound changes.