VPN flapping and false load balancer failures

The timing during database backups points to a traffic-path or capacity issue rather than a random tunnel failure.

First, investigate whether backup traffic is traversing the VPN. If backups between NIFCLOUD resources are routed over the office VPN instead of staying inside the private network, the tunnel becomes a bottleneck twice a day. The fix is to correct route tables so intra-cloud backup traffic uses the private network path, and to confirm that the backup source and destination do not depend on the office subnet.

Second, examine the second VPN tunnel. Two tunnels with the same remote subnet 192.168.10.0/24 and overlapping routes can cause route flapping, asymmetric routing, or duplicate security associations. The device may alternate between tunnels, or one tunnel may be preferred until a rekey event, then fail over badly. Check the route priority, tunnel configuration, and whether both tunnels advertise the same prefixes.

Third, check MTU and fragmentation. Large backup packets over a VPN can fragment or be dropped if the tunnel MTU is misconfigured, which may appear as a tunnel drop under load. Test with a reduced MSS/MTU and compare.

Finally, review VPN logs for rekey failures, phase 1 or phase 2 mismatches, and idle-timeout settings. A 2 to 5 minute drop is long enough to suggest a rekey or route convergence problem, not just a brief packet loss event.

The servers are running, so the health check is probably measuring the wrong thing or is too sensitive to a transient condition. During backups, disk I/O or CPU contention can delay the application's response beyond the health-check timeout. If the health check is a short TCP or HTTP check with a low timeout and a low failure threshold, a server that is busy but healthy can be marked down. The fact that exactly three servers are affected may also indicate that the check is hitting a shared dependency, such as a database or a backup-mounted volume, or that only some servers are on the backup path.

Recommended changes:

• Use a readiness endpoint that reflects the ability to serve traffic, not raw disk or backup status. If backups legitimately make a server slow, either schedule them outside the traffic peak or give the check a longer timeout and a higher failure threshold.
• Separate backup traffic from application traffic at the network level. Backups should not saturate the same path that health checks and client requests use.
• Review health-check interval, timeout, and unhealthy threshold together. A check that fails after one slow response is too aggressive for a backup window.
• Confirm that the load balancer checks the application port and path, not an unrelated port that happens to be open.
• Test the behavior deliberately. Run a backup on one server and observe health-check results before changing the whole fleet.

The design principle is that health checks should distinguish 'busy' from 'broken.' Marking a busy server down removes capacity exactly when the system needs it most.

A 30-minute client cache is a TTL problem, but simply lowering the TTL is not enough. The team should align DNS TTL, load balancer failover behavior, and client retry behavior.

• Lower the DNS TTL for the service name before a planned failover, but not so low that every client query becomes a load on the DNS service. A moderate TTL that matches the acceptable failover window is better than an extremely low TTL used all the time.
• Understand that some clients and resolvers ignore or extend TTLs. Do not assume that a TTL change takes effect immediately for every client. If the business requires fast failover, use a stable DNS name that points to a load balancer endpoint whose address can change, rather than expecting clients to re-resolve quickly.
• Keep the load balancer endpoint stable during failover. If the failover test changed the load balancer's address, clients that cached the old address will fail until their cache expires. Prefer a design where the DNS name remains constant and the load balancer handles backend failover internally.
• Set retry and connection timeout values on clients so they do not hold a stale connection indefinitely. A client that retries against the same cached address without re-resolving will not benefit from DNS changes.
• Document the failover sequence: lower TTL, verify propagation, shift traffic, confirm health, then restore TTL after a stability period.

The key judgement is that DNS is a discovery mechanism with caching behavior, not an instant switch. The failover design must tolerate clients that cache.