Prepared Query Returns Instances the DNS Interface Would Have Hidden

The two paths have different defaults for health filtering.

• The DNS interface implicitly filters to healthy instances: a plain payments.service.consul lookup only returns instances whose checks are passing.
• The prepared query data model has an OnlyPassing field that defaults to false. Traffic is therefore returned for critical instances too, which is exactly the 500s seen during the deploy.

So the migration silently removed a safety property that the applications had been depending on without knowing it.

Fix: rewrite the query definition with OnlyPassing: true (via PUT /v1/query/<id>) and confirm with a query against a service that has a failing instance. Note the interaction with failover: with OnlyPassing false, dc1 always returns a non-empty result set, so the failover policy never has a reason to fire. Enabling OnlyPassing both restores health filtering and makes the failover chain meaningful. Also budget for the fact that OnlyPassing can now produce an empty answer, which the frontend must handle.

Failover is evaluated per-request from the caller's perspective, using the datacenter the query is executed in.

1. Normal: the dc2 frontend resolves payments-lookup.query.consul. dc2 has 4 local instances, all passing, so the local result set is non-empty and the failover policy is never consulted. Traffic stays local. Nothing needs to be configured to achieve this beyond the query existing in dc2 with the same definition.
2. All four dc2 instances critical: with OnlyPassing: true, the dc2 result set becomes empty, so the failover policy advances to the next datacenter in the Datacenters list, i.e. dc1. Callers start crossing the WAN, which is the intended last resort but has latency and egress implications that should be called out.
3. Both datacenters empty: the query returns no answers. The client sees a negative response, so the application needs a fallback path rather than treating it as success.

If locality within a datacenter matters (many instances spread across racks/zones), add Near: _agent so results are sorted by round-trip time from the querying agent. That is a separate concern from datacenter failover and the two are commonly conflated.

The agents are the source of truth for their own registrations. Each client agent runs an anti-entropy sync against the catalog on a periodic interval (default 60s), pushing its local view of its node, services, and checks. Deleting entries directly from the catalog is therefore overwritten on the next sync - matching the observed one-minute reappearance.

Correct procedure, in order of preference:

• Deregister through the owning agent: consul services deregister -id=payments on that host, or stop the service so its check fails and let DeregisterCriticalServiceAfter on the check definition remove it after a defined grace period.
• Put a node into maintenance (consul maint -enable) when you want an agent's services to be excluded from discovery without touching registrations.
• Only use catalog-level deregistration when the agent is gone for good (host destroyed); otherwise you are fighting anti-entropy.

This also explains why catalog edits made during a snapshot restore or bulk-cleanup exercise can look like they "didn't take": the agents re-assert the previous state on the next sync.