Flipping default_policy to deny on a Live Datacenter

All three symptoms are the same failure seen through three interfaces: every request that no longer carries a privileged token now falls back to the anonymous token, because the client agents were never given one and the flip made the anonymous token powerless.

• DNS NXDOMAIN: DNS lookups are ACL-checked. The querying agent applies its default token, or the anonymous token if it has none. With default-deny and no rights on that token, the lookup returns no services - which callers see as NXDOMAIN, not a 403.
• Services not appearing: the agent's anti-entropy sync needs write access to its node and its services. 403 on that RPC is what the server logs show.
• KV 403: the checkout application is reading KV with no token (or an agent default token that lacks key:read on config/).

Confirmation, cheapest first: run consul catalog services -token=<management token> and compare to the same command without a token; read a client agent's log for the exact Permission denied line and the AccessorID it names; and use consul acl token read -self -token=<the token the app is using> to dump the effective policies. That tells you whether the token is missing entirely or merely under-scoped - a distinction that changes the fix.

Agent tokens: the idiomatic shape is one token per node built from node identity, e.g. consul acl token create -node-identity=web-01:dc1. That scopes the token to node web-01 in dc1, which is what the anti-entropy sync needs; services registered on that node are covered by the same identity. Where a team needs tighter blast radius (for example a single agent hosting services owned by different teams), create per-service identities (-service-identity=payments) and attach them explicitly, and remember the sidecar proxy is its own service - an Envoy sidecar for payments needs write access to payments-sidecar-proxy, which is a separate identity from payments.

Anonymous token: in a default-deny cluster that still answers DNS and serves a prepared query, the anonymous token must be given read-only discovery rights - service:read and node:read on the relevant prefixes, plus query:read on the prepared query name. Without those, every unauthenticated DNS lookup becomes NXDOMAIN, which is what broke first here.

Client-side default token: if you would rather not open the anonymous token, set acl.tokens.default on client agents so DNS lookups carry a read-only token. This is the tighter option and is preferred when the discovery surface shouldn't be world-readable.

The mistake was restarting servers into default-deny before any client had credentials. A workable sequence:

1. Secure a management credential first. Run consul acl bootstrap (or set acl.tokens.initial_management for the first boot only) and store the SecretID outside the cluster. Every subsequent token creation depends on it, and there is no recovery path if it is lost and no other management-capable token exists.
2. Create all tokens while default_policy is still allow. Nothing is blocked yet, so this is a no-op for traffic and lets you validate token scope with consul acl token read -self from each host.
3. Roll client agents first, one node per batch, writing acl.tokens.agent (and acl.tokens.default where needed) into config and restarting the agent. Verify each node with consul members -token=<token> and by confirming a service registration round-trips.
4. Roll servers last, one at a time, checking consul operator raft list-peers and waiting for a stable leader between nodes. The policy flip only takes effect when the servers reload.
5. Verify the read paths explicitly: a DNS lookup from an agent, the prepared query, a KV read, and one Connect call through a sidecar - each fails under a different ACL rule and a green catalog does not prove any of them work.

Note that unsealing a lockout after the fact is painful: if no management token exists and acl.tokens.initial_management was never set, you are editing server config files in place, which is why step 1 is non-negotiable.