Global Traffic Manager Keeps Returning a Degraded Data Center
BIG-IP DNS 16.1 hosts wide IP www.example.com with two pools: dc1_pool (ten members, 10 Gbps of usable capacity) and dc2_pool (ten members). The wide IP uses Topology load balancing: source region Europe resolves to dc1_pool, North America resolves to dc2_pool, and the fallback is Global Availability. TTL is 30 seconds. Monitors are HTTP GET /health with a 5-second interval, 16-second timeout, and 3 retries. A rack power failure in DC2 takes down eight of ten members. The two survivors pass the health check but can handle only about 1,200 requests per second combined; DC2 is still receiving about 9,000 requests per second, roughly 40% of the 22,500 total. North American users report 30% HTTP 5xx responses and 8-second latency, while European users are unaffected. BIG-IP DNS continues to return DC2 addresses for North American resolvers. Some public resolvers appear to cache the answer for 300 seconds even though the wide IP TTL is 30. The operations team has no ability to change client DNS settings and wants a fix that does not require a maintenance window.
1. Why does BIG-IP DNS continue to return DC2, and which statistics would you check to confirm the explanation?
Read the worked answer
BIG-IP DNS is doing what it was configured to do. Topology returns dc2_pool for North American resolvers, and Global Availability only moves to another pool when the current pool is unavailable. A pool is unavailable when all members fail their monitor, not when capacity is degraded. Two members still pass the HTTP /health check, so the pool remains available and the wide IP keeps returning DC2 addresses. The monitor is also too shallow: a static /health response can succeed while the application is saturated. The 9,000 requests per second arriving at a 1,200-request-per-second pool explains the 5xx and latency.
The 30-second TTL is not the primary cause; the primary cause is that no health signal represents capacity. Some resolvers caching for 300 seconds will extend the bad answer even after GTM changes its decision, but that is a secondary effect. Confirm by checking wide IP and pool statistics, member monitor status, and the topology records for the affected source region.
2. What immediate operational action and what structural changes would you make?
Read the worked answer
Immediate control: disable dc2_pool or set its members to disabled so GTM stops returning it, and let the fallback send North American users to DC1. This is an operational action, not a configuration change, and it does not need a maintenance window. If DC1 cannot absorb the load, use a ratio or topology record to split traffic only to the surviving DC2 members at a rate they can sustain, rather than 40%.
Structural controls: replace the shallow HTTP monitor with one that checks a dependency or reports capacity, or use a dynamic ratio or QoS method driven by SNMP or iQuery metrics so pool weight reflects real load. Set an availability requirement so a pool with fewer than a defined number of healthy members is treated as unavailable. Add a prober pool if path quality matters. Keep the topology fallback explicit so there is a deterministic answer when the preferred pool is removed. Test the new monitor in alarm-only mode before enforcing it.
3. How do you deal with resolvers that cache the old answer longer than the TTL?
Read the worked answer
You cannot force a public resolver to honor a 30-second TTL. Some resolvers clamp or cache for 300 seconds, so a GTM change may take up to five minutes to reach end users. Design for that: keep TTL low, but do not rely on it as the only failover mechanism. Use GTM to remove a bad pool quickly, and use LTM inside each data center for fast member-level failover. If the application can tolerate it, return a CNAME to a per-data-center name with a short TTL so you can change the target without changing the wide IP. Consider anycast or a third-party DNS provider only if the business requires sub-minute global failover.
For this incident, the practical answer is to disable DC2 at the wide IP level, accept that some users will see the old answer for up to five minutes, and communicate that window. After the incident, review whether the topology records and monitor thresholds match the real capacity of each pool.