Federated Services Fail After a Data Center Failover

The pattern points to a URL registration problem, not a capacity or network problem, because the remaining DC-B machine is proven reachable through the F5 from outside.

A federated server is registered in the portal with two URLs: the server URL (what clients use to draw services) and the administration URL (what the portal itself uses to manage the site, read its services, and check health). Federation is a portal-driven relationship: when a user opens a map service in Map Viewer, the portal validates the item and often proxies or re-checks the service through the registration. If the administration URL was entered as a machine name such as https://gis1.dc1.internal:6443/arcgis/admin instead of the F5 name, the portal can only reach the federated site while that specific DC-A machine is alive - even though the public F5 URL still routes to DC-B. That explains all three symptoms: clients and anonymous REST work (F5 pool has DC-B), hosted layers work (they belong to the hosting server site, registered separately), and the portal cannot validate the federated services.

Evidence to collect:

1. Portal Directory > Servers > the federated site: record both the admin URL and server URL verbatim.
2. From each portal machine, nslookup and curl the admin URL to prove reachability (or failure).
3. Portal logs around the outage for messages naming a hostname rather than the F5 name.
4. Server Manager > Site > WebContextURL on the federated site, to see what URL the server advertises.
5. The server-side portal registration (Server Manager > Site > Portal) to confirm it points at the F5 portal name.

The flicker is a side effect: the portal's health probe alternates between a cached healthy result and a failed live probe, so status should not be trusted until the registration is corrected.

Correct the registration so that every cross-component reference resolves through the load balancer, never through a machine name:

• Federated server administration URL - set to https://gis.city.gov/server/admin. This is what the portal uses to manage and validate the site; it must be reachable from every portal node and must survive the loss of any one GIS machine.
• Federated server server URL - set to https://gis.city.gov/server. This is what the portal stores on items and what clients receive; a machine name here produces broken items even when the service is running.
• WebContextURL on the GIS site - set so the server generates REST and token URLs with the public F5 name behind the reverse proxy. Without it, service directories and token endpoints advertise internal hostnames.
• Server-side portal registration (Server Manager > Site > Portal) - point at the F5 portal name so the GIS tier talks to the load-balanced portal, not to portal1 only.
• Hosting server site registration - apply the same rule; hosting servers are frequently registered with a single machine name during setup and quietly become a single point of failure.

Two cautions. Changing WebContextURL requires a service restart and can invalidate saved URLs in existing items, so plan it in a window and re-test a sample of items afterward. Second, confirm the F5 pools contain nodes from both data centers and that no persistence (sticky sessions) is required for portal traffic, since a portal node that is drained should not orphan user sessions. After the changes, repeat the failover drill: stop one portal node and one GIS node and verify that sign-in, map drawing, and publishing all still work.

A TCP/443 check only proves that something accepted a socket on that host; it says nothing about whether the ArcGIS Server site, its services, or the portal-to-server relationship are usable. In this incident, a TCP monitor would happily mark the DC-B machine as up (which it was) while giving no signal about the portal's ability to reach the site's administration endpoint. That gap is exactly why the team was surprised: infrastructure monitoring said "healthy" while Map Viewer said "service unavailable."

Changes I would make:

1. Application-layer check - issue an HTTPS GET against a lightweight, stable endpoint such as /server/rest/info?f=json and assert HTTP 200 plus a parsable JSON body. Avoid /rest/services as a health target; on a large site it can be expensive and slow under load.
2. Probe from the consumers, not only the pool - run the same check from each portal node and from the F5 itself, so a routing or DNS asymmetry between tiers is detected.
3. Explicit portal-to-server check - for the federated site, confirm that a portal-issued token can be validated, or at minimum that the administration endpoint returns a 401 challenge rather than a connection failure.
4. Sane thresholds - three consecutive failures before alerting to avoid flapping, and alert on the load-balancer pool membership changing.
5. Alert on the portal side too - Portal Directory server status should feed monitoring; a server shown as Unavailable for more than a few minutes should page someone.

The general principle: monitor the transaction the user actually performs (portal to server to data), not the port underneath it.