Diagnosing intermittent 5xx errors after a VPC redesign
An engineering team hosts a three-tier app in a single Alibaba Cloud VPC: an SLB in vSwitch A (10.0.1.0/24), application ECS instances in vSwitch B (10.0.2.0/24), and an ApsaraDB for RDS instance in vSwitch C (10.0.3.0/24). After a redesign, users report intermittent HTTP 502/504 errors from the SLB, but only under load and only for some requests. Direct curl tests from an app server to the RDS endpoint succeed. The team recently tightened a security group on the app tier to allow inbound traffic only from the SLB's vSwitch CIDR and added a new Network ACL. Monitoring shows app CPU is healthy. Some app instances occasionally fail SLB health checks and are removed, then re-added minutes later.
1. Given the symptoms, what is the most likely cause of the intermittent 502/504 errors, and how would you confirm it?
Read the worked answer
The pattern - intermittent, load-dependent 5xx, healthy CPU, and instances flapping in and out of health checks - points to a network/health-check path problem rather than application logic. The prime suspect is the recently tightened security group or the new Network ACL blocking the SLB's health-check probes or back-end traffic for some instances.
Reasoning:
- Intermittent + instance-specific suggests only some instances are affected, consistent with a rule that does not cover all back-end IPs or the health-check source correctly.
- 502/504 from the SLB means the SLB could not get a valid response from the back end - either the connection was refused/blocked or it timed out.
- Health-check flapping is the tell: instances that fail probes get removed, then re-added when a probe happens to pass, producing exactly this on/off behavior.
How to confirm:
- Check the SLB health-check status per back-end and the configured health-check port/protocol.
- Inspect the security group rules on the app tier - verify the allowed source matches the SLB health-check and forwarding source, not just the vSwitch CIDR if the probe originates elsewhere.
- Inspect the Network ACL rules (both directions, ephemeral port ranges) for denies affecting the SLB-to-app path.
- Use VPC flow logs / traffic mirroring or packet capture on a flapping instance to see whether probes arrive and whether replies leave.
Confirming the blocked path before changing app code avoids chasing a phantom application bug.
2. A colleague proposes simply widening the app-tier security group to allow all traffic from 0.0.0.0/0 to stop the errors. Evaluate this proposal and describe a safer approach.
Read the worked answer
Widening to 0.0.0.0/0 would likely mask the symptom but is the wrong fix:
- It removes the security boundary the team just added, exposing the app tier to the internet and any lateral movement.
- It does not address the ACL. If the new Network ACL is the actual blocker, opening the security group changes nothing and the flapping continues.
- It hides the real cause, making future diagnosis harder.
A safer approach:
- Identify the exact source of SLB health checks and forwarded traffic and allow only that (least privilege), rather than a broad CIDR.
- Fix the ACL deliberately - add explicit allow rules for the SLB-to-app path in both directions, including the ephemeral port range for return traffic, and keep default-deny elsewhere.
- Document the intended flow (SLB → app → RDS) and align both security group and ACL rules to it.
- Validate by watching health-check status and error rates under load before declaring done.
Least-privilege rules scoped to the real traffic path fix the errors without opening the tier to the world.