Five-second first request after an egress NetworkPolicy rollout

It is a TCP connect timeout, not DNS. The twelve milliseconds of search-domain NXDOMAINs in the same capture are real but negligible - they also prove DNS is answering promptly. The five seconds is the SYN to 198.51.100.21 going unanswered: a NetworkPolicy drop is a silent DROP, not a REJECT, so the client blocks until its own connect timeout, then moves to the next address in the list, which happens to be inside the allowed /26.

Reconstruct the arithmetic: api.psp.example.com has three dots, fewer than ndots:5, so glibc fires the name against three search domains before going absolute - six queries counting A and AAAA. That finishes in 12 ms. It changes nothing about the five-second wall clock.

Confirm before changing anything:

• kubectl exec a debug pod and run nc -vz -w 2 198.51.100.21 443 (hangs, then times out) vs 203.0.113.36 (connects).
• Read the policy back: kubectl get networkpolicy -n payments -o yaml - 198.51.100.21/22 are outside 203.0.113.0/26.
• The same addresses connect fine from a namespace with no default-deny, which rules out provider geo-blocking or an outage; a provider problem would more likely RST or return an HTTP error than blackhole.

Immediate fix: extend the egress to.ipBlock.cidr list to cover the full published range - add 198.51.100.16/28 (or whatever the provider documents) alongside 203.0.113.0/26, port 443 only. Do not widen to 0.0.0.0/0; that silently reverts the entire default-deny design, and the next incident will not have a policy to blame.

Durable fix: stop encoding IPs. Vanilla Kubernetes NetworkPolicy has no DNS-name field, so FQDN allowlisting requires a CNI that supports it (Cilium toFQDNs) - state that constraint rather than pretending the built-in object can do it. The CNI-agnostic pattern is an egress proxy or gateway in the namespace: allow 443 only to the proxy, let the proxy do DNS-based allowlisting, and provider IP churn becomes a proxy-config change rather than a cluster rollout.

Add a canary that dials the provider from each namespace every minute and alerts on both latency and failure, so a blackholed address shows up as a metric instead of a customer report. Separately, ndots: 2 is reasonable hygiene - it removes six wasted queries per connection - but be honest that it would not have changed the five seconds.

Three candidates, in order of likelihood.

1. It uses hostNetwork: true. NetworkPolicy is enforced by the CNI on the pod's veth; host-network pods have no veth in the pod network and bypass policy entirely, so the default-deny never applies. Check kubectl get pod <name> -o jsonpath='{.spec.hostNetwork}'.

2. A pre-existing established flow. Policy is evaluated on new connections; an established conntrack entry keeps forwarding after the policy is applied, until the socket closes. Verify on the node with conntrack -L | grep 198.51.100 while the pod is running. The tell is that a fresh connection (curl --no-keepalive or a new process) fails while the long-lived one still works.

3. Different DNS answers. If the pod's resolver returned only the allowed addresses (round-robin ordering, or a stale cache), it never dials the blocked one. Confirm with getent hosts api.psp.example.com from inside the pod.

How to separate them: from the suspect pod, force a brand-new connection to 198.51.100.21 with a short timeout. Instant success plus hostNetwork: true means bypass; success only on the old socket means conntrack; a failure means the pod is simply getting better DNS answers and the problem is still there, just latent.