Catalog-API registrations leave dead payment instances in DNS
Payments runs 60 instances in an Auto Scaling group. A deployment script registers each instance by calling PUT /v1/catalog/register on a server's HTTPS API, using an ACL token with service:payments:write and node:write on the instance's node name. Each payload includes a check: {"Name":"http","HTTP":"http://10.4.x.x:8080/health","Interval":"10s","Status":"passing"}. Scale-in works, but termination never calls any deregistration endpoint. Over three weeks, DNS answers for payments.service.consul grow to 60 addresses, roughly 12 of which are dead, and Envoy upstreams for checkout see intermittent connection refused. consul catalog nodes -service=payments lists all 60 nodes as passing. The team also reports that deregister_critical_service_after is set on the check in the payload but nothing ever deregisters.
1. Why do terminated instances remain in DNS as passing, and why does the health check never flip them to critical?
Read the worked answer
The registrations bypass the agent. /v1/catalog/register writes directly into the catalog; there is no client agent owning that service, so there is no anti-entropy loop reconciling the catalog with reality and no check runner executing the HTTP probe. The check definition is stored, but nothing ever calls http://10.4.x.x:8080/health.
Worse, the payload hard-codes Status: "passing". The catalog therefore asserts that the service is healthy and never revises that assertion. DNS service lookups only include instances whose checks pass, so all 60 are returned, including the 12 dead ones.
deregister_critical_service_after is an agent-side behavior attached to checks the agent actually executes; it has no effect on catalog-written entries. And because termination never calls the deregister endpoint, the records persist indefinitely.
This is the classic catalog-API trap: it is a synchronization primitive for external systems, not a self-maintaining registration mechanism. It is appropriate when another system (Kubernetes sync, a cloud controller) owns lifecycle and keeps the catalog in sync. Here nothing owns it.
2. What do you run right now to clean up, and what are the risks of each cleanup action?
Read the worked answer
1. Snapshot first. consul snapshot save pre-cleanup.snap so a bad deregistration is reversible.
2. Reconcile against the source of truth. List catalog nodes for the service (consul catalog nodes -service=payments) and join against the ASG's current instance inventory. Anything in the catalog but not in the ASG is stale. Do not trust the passing status, and do not infer health from the Consul UI alone.
3. Deregister by service ID, not by node. PUT /v1/catalog/deregister with {"Node":"<node>","ServiceID":"payments-i-0abc..."}. Deregistering at the node level removes every service and check on that node, which is dangerous if the node hosts sidecars or other services, or if the node is still alive but only this instance is stale.
4. Watch the blast radius. Deregistration of a service that still exists in the catalog on another node is harmless; deregistering a live node's entry removes it from DNS until re-registered. Confirm ASG termination order and ensure you are not removing an instance still receiving traffic.
5. Re-query DNS and health API (/v1/health/service/payments?passing) to confirm the address set matches the ASG. Then fix the root cause the same day, because the next scale-in recreates the problem.
3. Redesign registration so this self-heals. What components change and which check settings matter?
Read the worked answer
Move registration to a local agent. Run a Consul client agent on each instance (or a DaemonSet in Kubernetes) and register with /v1/agent/service/register (or consul services register). The agent now owns the service: anti-entropy syncs it to the catalog, the agent executes the HTTP check on the declared interval, and status transitions are real.
Use lifecycle hooks for decommission. On ASG termination, run consul services deregister -id=payments-<instance-id> before shutdown. Belt and braces: set leave_on_terminate = true on clients and set deregister_critical_service_after = "5m" on the check so an abruptly killed instance is reaped after checks go critical.
Make service IDs unique (payments-<instance-id>) so registrations never overwrite each other and deregistration is surgical.
Token design. The node's token needs node:write scoped to that node plus service:payments:write; use the agent's built-in service identity rather than a shared token, so a compromised instance cannot register other services or impersonate one behind an intention.
Verify with the health API, not the catalog. Routing should use /v1/health/service/...?passing or DNS, both of which honor check status. Add an alert when the count of passing instances deviates from the ASG's in-service count by more than one, which catches both stale entries and registration failures.