Load balancer health checks and cross-region DNS failover
Fabrikam runs a public web service behind an OCI flexible load balancer in VCN 10.30.0.0/16. The LB uses a regional public subnet 10.30.1.0/24 and a 100 Mbps flexible shape. The backend set contains four application servers in private subnet 10.30.2.0/24, listening on HTTP 8080. The health check is HTTP GET /health every 10 seconds, timeout 3 seconds, unhealthy threshold 3. Users report intermittent 502 responses, and the LB marks one backend unhealthy for a few minutes at a time. Application logs show successful requests from users, but no health-check requests from the load balancer on the affected server. The app subnet security list allows ingress TCP 8080 only from 10.30.2.0/24, and each app server has an NSG that allows 8080 from the LB's NSG. The listener is HTTPS 443 with a certificate for www.example.com. DNS is a CNAME to the LB public IP with TTL 300. The team wants to fix the health check and add cross-region failover with a 5-minute RTO.
1. Diagnose why the LB marks a backend unhealthy even though the app NSG allows the LB NSG. What exact rule change and verification steps do you recommend?
Read the worked answer
The NSG is not the only control. OCI evaluates both the NSG and the subnet security list. The app subnet security list allows TCP 8080 only from 10.30.2.0/24, but LB health-check traffic originates from the LB subnet 10.30.1.0/24. The NSG allows the LB NSG, but the security list drops the probe before it reaches the application, which explains why no health-check requests appear in the app logs.
Add an ingress rule to the app subnet security list allowing TCP 8080 from 10.30.1.0/24, or narrow it to the LB subnet and port. Keep the NSG reference to the LB NSG as the workload-level control. Verify that /health returns HTTP 200 directly from the app and does not redirect to HTTPS; a redirect can fail an HTTP health check. Then confirm the LB backend set shows healthy and use VCN flow logs to see accepted probes from the LB subnet. If probes still fail, check the 3-second timeout and the app's response time.
2. How would you implement cross-region failover with a 5-minute RTO, given the current DNS record?
Read the worked answer
A CNAME to a single regional LB public IP cannot provide cross-region failover; if the region or LB is unavailable, DNS still points clients to a dead endpoint. Keep the regional LB for in-region AD fault tolerance, but put a DNS-based failover layer in front.
Use OCI Traffic Management or an equivalent authoritative DNS service with a failover steering policy. Define a primary endpoint for the region-one LB and a secondary endpoint for a region-two LB, attach health checks to both, and set a low TTL such as 60 seconds. The 5-minute RTO is achievable if health-check intervals and failure thresholds are tuned so that detection plus DNS propagation stays under five minutes. Do not rely on the LB's own regional failover for a region-wide event. Test by disabling the primary health check and confirming that resolvers receive the secondary answer within the RTO.
3. What should the team check about SSL termination and client IP visibility before changing the listener?
Read the worked answer
The listener terminates HTTPS at the LB, so the application sees the LB as the TCP peer unless the LB forwards client information. For HTTP/HTTPS listeners, configure the LB to add the X-Forwarded-For header and make the application trust it only from the LB. If the application requires the original client IP at the TCP level, or requires end-to-end TLS without LB termination, use a TCP listener with SSL pass-through instead; that changes certificate management and health-check options.
Also verify that the backend health check still uses the correct protocol and port after any listener change. Terminating TLS at the LB does not affect the HTTP health check on 8080, but pass-through TLS would require a TCP health check. Document which layer owns the certificate and how the app enforces trusted proxy headers.