CKA (Other exams): ClusterIP resolves but times out: a NetworkPolicy whose namespaceSelector matches nothing

Work from the outside in, and use the shape of the failure as evidence:

• DNS works because dev's egress policy permits 53 to kube-system, and CoreDNS is not in prod. So name resolution tells you nothing about reachability of api.
• The connection times out rather than being refused. A refusal means the packet reached a host stack with nothing listening - wrong targetPort, or an explicit REJECT. A silent drop is what a NetworkPolicy denial looks like: the CNI programs a drop in the pod's ingress path, the SYN is discarded, and the client retries until curl -m 5 gives up.
• Endpoints are populated and pods are Ready, so this is not a selector typo, not a readiness failure, and not a kube-proxy/iptables problem - if kube-proxy were broken, the ClusterIP itself would not be programmed and you would typically see a refusal or no route.

So the packet is being DNATed to a real pod IP and dropped on arrival: the prod ingress policy. Confirm with:

1. kubectl exec -n ingress-nginx <controller-pod> -- curl -m 3 http://api.prod.svc.cluster.local - if this also hangs, the fault is not specific to dev.
2. kubectl get netpol -n prod -o yaml - note that deny-all-ingress makes the namespace default-deny, and allow rules are additive, never subtractive.
3. kubectl get ns ingress-nginx --show-labels - check whether the name: ingress-nginx label actually exists.

Check the label. Since Kubernetes 1.21 every namespace automatically carries kubernetes.io/metadata.name: <name>, but it does not automatically carry name: <name>. A namespaceSelector matching name: ingress-nginx therefore selects an empty set, the allow-ingress rule grants nothing, and deny-all-ingress is the only rule that takes effect for both sources.

That explains both symptoms from one cause:

• The ingress controller's upstream connection to the pod IP:8080 is dropped, so nginx waits on its proxy_read_timeout (30s here) and returns 504 Gateway Timeout. A 503 would have meant nginx had no endpoints at all - different fault, different fix.
• The dev debug pod hits the same drop and its curl hangs.

Note that the traffic never reaches the application: api pods are Ready and serving /healthz, and there is nothing to see in their logs. That is a strong hint that the drop is in the network layer, not in the app. Verify the hypothesis cheaply before editing anything: temporarily add the correct label with kubectl label ns ingress-nginx name=ingress-nginx and re-run both curls - if both recover, the diagnosis is confirmed. Then remove the ad-hoc label and fix the policy properly, because relying on an out-of-band label is exactly how this broke in the first place.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-ingress
  namespace: prod
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: ingress-nginx
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: dev
    ports:
    - protocol: TCP
      port: 8080

Justification:

• port: 8080 is the container port, not the Service port. NetworkPolicy is enforced at the pod's network interface, after kube-proxy has DNATed 80 → 8080. Writing port: 80 is the most common way this policy gets built wrong and silently does nothing.
• Two list items in from are ORed. A single item containing both namespaceSelector and podSelector would AND them, granting only pods that are in those namespaces and match the labels - not what we want.
• The built-in kubernetes.io/metadata.name label removes the need to label namespaces by hand and cannot drift.
• deny-all-ingress stays. Policies are additive; there is no need - and no way - to delete a deny rule from another policy object. Do not remove the default deny, it is doing useful work for every other pod in prod.
• Leave dev's egress policy alone; it already permits 53, and DNS proved it. If dev were also default-deny on egress to prod, this policy would look correct and still fail - always check both directions.