Intermittent DNS failures with toFQDNs egress policy
A payments namespace runs 40 pods. A CiliumNetworkPolicy selects all pods in the namespace with default-deny egress, then allows UDP and TCP 53 to kube-dns and a separate rule with toFQDNs: matchName: api.stripe.com. After rollout, about 5% of requests fail with 'no such host', and a few succeed but connect to an old Stripe IP that is no longer returned by DNS. The policy has an egress rule allowing port 53 to kube-dns but that rule has no rules.dns section. Cilium 1.15 is running with the DNS proxy enabled. cilium monitor --type policy-verdict shows DNS replies from kube-dns are allowed, but some A records never appear in the FQDN cache. The application uses a 30-second DNS TTL and makes short-lived HTTPS connections. The team wants to keep egress deny-by-default and avoid broad CIDR allow rules.
1. Why does allowing port 53 to kube-dns not make toFQDNs work reliably? What policy element is missing?
Read the worked answer
Allowing UDP and TCP 53 to kube-dns only permits the packets; it does not make Cilium inspect the DNS answers. toFQDNs depends on the DNS proxy learning names and addresses. The proxy is invoked when the policy rule contains a rules.dns section. Without rules: dns: - matchPattern: '*', Cilium treats DNS as ordinary L3/L4 traffic, so the FQDN cache is not populated reliably. The intermittent 'no such host' failures happen when the pod's DNS query is allowed but the resulting IP is not in the cache, and the later egress connection is denied because the policy has no matching toFQDNs entry. The stale-IP case is the opposite failure mode: an old cached address remains usable after DNS changes, often because the FQDN cache lifetime is longer than the record TTL.
2. Write the corrected egress policy structure and explain how DNS proxy and FQDN cache interact.
Read the worked answer
Correct structure: keep egress default-deny, then add one rule that allows DNS to kube-dns and includes DNS proxy rules, and a second rule for the FQDN.
endpointSelector:
matchLabels:
io.kubernetes.pod.namespace: payments
egress:
- toEndpoints:
- matchLabels:
io.kubernetes.pod.namespace: kube-system
k8s-app: kube-dns
toPorts:
- ports:
- port: '53'
protocol: UDP
- port: '53'
protocol: TCP
rules:
dns:
- matchPattern: '*'
- toFQDNs:
- matchName: api.stripe.com
The first rule must come before the FQDN rule so DNS queries are allowed and proxied. The DNS proxy then inserts observed A/AAAA records into the FQDN cache, and the toFQDNs rule permits connections to those addresses. matchPattern: '*' is broad for DNS names but does not allow arbitrary egress; the toFQDNs rule still gates the actual connection.
3. How would you diagnose the stale-IP symptom and reduce its impact without weakening egress control?
Read the worked answer
Diagnose with cilium-dbg fqdn cache list (or cilium fqdn cache list depending on version) to see the cached names, IPs, and expiry; cilium-dbg monitor --type dns to confirm queries are passing through the proxy; and cilium-dbg policy trace from the pod identity to api.stripe.com:443. Compare the cache TTL with the 30-second DNS TTL. If the cache expiry is much longer, lower the FQDN minimum TTL via the Cilium config, for example --tofqdns-min-ttl=30, and roll the agents. Do not solve the stale-IP issue by adding broad CIDR egress rules; that defeats the control. After the change, force a DNS lookup from a test pod, confirm the cache refreshes, and watch for policy-denied verdicts during the TTL transition. If the application caches connections longer than the DNS TTL, that also needs to be addressed at the application layer.