I85PREP
⌘ K
BR-ANAC-DOV-MOD-1 study materials
BR-ANAC-DOV-MOD-1 study resources

Overlay Network Cannot Resolve Service Name Across Nodes

BR-ANAC-DOV-MOD-1 · Put your knowledge to work in context.

All case studies

Overlay Network Cannot Resolve Service Name Across Nodes

A swarm has three manager nodes and two workers. A frontend service (2 replicas) and a backend service (2 replicas) share an overlay network appnet, created with docker network create --driver overlay --attachable appnet. Frontend containers intermittently fail to resolve backend and log no such host, roughly 20% of requests, while other requests succeed. Both services are on appnet. A recent change added --endpoint-mode dnsrr to backend. One worker was rebooted yesterday and rejoined. docker service ps backend shows all tasks running. Direct IP connections to backend tasks work. The team also has a legacy standalone container on one worker that must reach backend by name.

1. Analyze the likely causes of intermittent name resolution failure given the DNSRR change and the rebooted worker.

Read the worked answer

With --endpoint-mode dnsrr, the service name resolves to the task IPs directly rather than a VIP. The embedded DNS server returns the current set of task IPs. Intermittent no such host suggests the DNS answer set is stale or incomplete, not that the service is down.

Two suspects fit the evidence:

  1. The rebooted worker's overlay state. After a reboot, the node may have rejoined the swarm but its overlay network endpoints or the embedded DNS entries may be partially reconciled. Tasks report running, but name resolution from other nodes can lag or miss endpoints until the overlay is fully re-established. A node leave/rejoin or service reschedule often clears it.
  2. DNSRR with changing task IPs. Because DNSRR returns task IPs, clients that cache DNS aggressively or hold connections can see failures when tasks are replaced. A 20% failure rate is consistent with one of two replicas being unreachable by name from certain nodes.

Direct IP connections working rules out a general network partition; the problem is name resolution, pointing at the embedded DNS and overlay endpoint state rather than routing.

2. Propose a diagnostic sequence and a fix that also lets the legacy standalone container resolve `backend`.

Read the worked answer

Diagnose

  1. From a frontend container, run getent hosts backend or nslookup backend repeatedly; note whether the returned IP set is missing a task.
  2. docker service inspect backend --format '{{json .Endpoint}}' and docker service ps backend to list task IPs; compare against DNS answers.
  3. On the rebooted worker, docker network inspect appnet and check docker info for swarm state; look for overlay errors in dockerd logs.
  4. docker exec into both frontend replicas; if only one node's frontend fails, the fault is node-local.

Fix

  • If the rebooted worker is the culprit, drain and rejoin it (docker node update --availability drain, then rejoin), or force-reschedule the affected tasks so overlay endpoints are rebuilt.
  • If DNSRR caching is implicated, either revert to VIP endpoint mode (default) for stable service-name resolution, or keep DNSRR and ensure clients do not cache DNS beyond TTL and reconnect on failure.

Legacy container access The standalone container is not a swarm service, so it must attach to the attachable overlay network: docker run --network appnet .... Only then does it get embedded DNS for backend. Confirm the container is on the same worker that can reach the overlay, and that appnet is indeed attachable (it is, per the create flags).