GSLB keeps directing users to a site whose application is down
A travel company runs an active-passive GSLB deployment for book.example.com. DC1 at 198.51.100.10 is priority 1, and DC2 at 203.0.113.20 is priority 2. The GSLB vserver uses the Failover method, a 30-second DNS TTL, and MEP is enabled between the sites. Each GSLB service points to the site's public VIP and uses a TCP monitor on port 443. Because the ADC owns the VIP and completes the TCP handshake, the monitor stays up even when every backend service is down. During a planned application upgrade in DC1, the application servers were stopped for 25 minutes while the ADCs and VIPs stayed up. GSLB continued to return DC1 to about 80% of resolvers, and users saw HTTP 503 pages. After the upgrade, the team wants automatic failover for real application failures but wants to avoid flapping when only one of six app servers is down. They also want users who already hold a GSLB site-persistence cookie for DC1 to be moved when DC1 is truly unavailable.
1. Why did GSLB not fail over, and what monitor design fixes it?
Read the worked answer
GSLB did not fail over because its health signal was measuring the wrong thing. The GSLB service monitor was TCP/443 against the public VIP. The ADC completed the TCP handshake even though the application servers behind it were stopped, so the GSLB service remained UP and the Failover method kept DC1 as the preferred site.
Replace the TCP monitor with an HTTP-ECV or HTTP monitor that requests a site health endpoint and expects HTTP 200. Point it at the VIP and include the correct Host header. The endpoint must reflect whether the site can actually serve users, not merely whether the ADC is listening. With an interval of 5-10 seconds and three retries, a total application outage will mark the GSLB service DOWN and fail traffic to DC2.
For the one-of-six-server case, the site health endpoint should aggregate capacity. If the site is designed to serve with five of six servers, the endpoint should still return 200. If the site cannot serve below a threshold, the endpoint should return 503. This avoids flapping the entire site for a single failed backend. Bind the monitor to the GSLB service, not only to the local load-balancing vserver, and verify with show gslb service and stat gslb vserver.
2. How should site persistence be configured so existing cookie users move during a site outage without losing affinity in normal operation?
Read the worked answer
Site persistence should be configured on the GSLB vserver so that a user who is directed to DC1 continues to use DC1 while DC1 is healthy. The cookie is an affinity hint, not a replacement for health. When the GSLB service for DC1 is marked DOWN, the GSLB decision must select DC2, and the DR site must issue a new site cookie rather than honoring a stale cookie that points to an unavailable site.
Use HTTP cookie-based site persistence if the application is HTTP and the GSLB deployment supports it; otherwise use source-IP site persistence and accept the NAT limitations. Set a persistence timeout that matches the application session, not an excessively long value. Test the behavior explicitly: establish a session through DC1, confirm the cookie, stop the DC1 application so the GSLB service goes DOWN, and verify that a new request is served by DC2. Then restart DC1 and confirm that failback does not cause session churn.
Remember that DNS TTL is only a hint. A 30-second TTL does not override a client or resolver that caches longer. Do not rely on TTL alone to move existing users; use health-driven GSLB selection and a short, tested persistence timeout.
3. What would you change to prevent flapping, and what evidence would you collect during the next failover test?
Read the worked answer
Prevent flapping by separating site health from individual server health. Use a site-level health endpoint that returns success while the site has enough capacity and failure only when it cannot serve. Tune the monitor so a single transient timeout does not mark the site DOWN, for example interval 10 seconds with three retries. For planned maintenance, administratively disable the GSLB service or put the site in a controlled maintenance state rather than relying on a race between application shutdown and monitor timing.
For failback, avoid immediate automatic return if the business wants a validation window. Keep DC1 at a lower priority or administratively disabled until application and database replication checks pass, then re-enable it. Document the manual step so it is not forgotten.
During testing, collect show gslb site, show gslb service, stat gslb vserver, MEP statistics, monitor state transitions, and DNS answers from multiple resolvers. Capture the time from application stop to GSLB DOWN, the time for DNS to return DC2, and the behavior of a client holding a DC1 site-persistence cookie. Repeat for a single backend failure to prove that the site stays up.