CKA (CNCF): NodePort with Local Policy Behind a TCP Load Balancer

Because externalTrafficPolicy: Local installs forwarding rules only on nodes that already host a ready endpoint for that Service.

How the packet path differs:

• On worker-1/2/3: kube-proxy has programmed a NodePort chain that DNATs straight to the local web pod. The connection succeeds, and because there is no second hop, the pod sees the genuine client source IP (no SNAT). The LB's TCP check on 30080 passes.
• On worker-4/5/cp-1: there is no local endpoint, so kube-proxy programs the NodePort to REJECT rather than forward. The TCP SYN never reaches a backend; the LB's health check gets connection refused and marks the node down. That is why users landing on those nodes see the same refusal - they are hitting the same REJECT rule.

This is by design, not a bug. kube-proxy also exposes a separate healthCheckNodePort (visible as spec.healthCheckNodePort on the Service, listed in kubectl get svc web -o yaml), which returns 200 only when the node has local endpoints and 503 otherwise. Your LB is effectively performing the same test by probing 30080 directly, which is why its verdict lines up so cleanly with the endpoint distribution.

Verify:

kubectl get endpointslices -l kubernetes.io/service-name=web -o wide
kubectl get svc web -o jsonpath='{.spec.healthCheckNodePort}{"\n"}'

The endpoints are only on three nodes, which is exactly the set that passes. Nothing is misconfigured - the topology and the policy simply disagree.

The constraint is structural: with Local, a node can only serve traffic if it hosts a ready endpoint. Preserving the client IP means you cannot fall back to Cluster, so you must change where the endpoints are, not the policy.

Option A - switch to externalTrafficPolicy: Cluster (one line, wrong answer).

kubectl patch svc web -p '{"spec":{"externalTrafficPolicy":"Cluster"}}'

All six nodes pass the check because any node now forwards to any pod. But the forwarding node SNATs the traffic, so pods on worker-1/2/3 see a node IP, not the client IP. This breaks the stated requirement. Mention it, do not do it.

Option B - make every node host a web endpoint. Scale to 6 with required pod anti-affinity on kubernetes.io/hostname, and add a toleration if you also want cp-1. Local then works on all six nodes and client IPs are preserved end to end. Cost: 6 pods instead of 3, and cp-1 needs a toleration for its control-plane taint.

Option C - put a DaemonSet proxy in front (recommended). Run ingress-nginx as a DaemonSet with hostPort 80/443 (or a NodePort Service with externalTrafficPolicy: Local) on every node. Every node now has a local endpoint for the proxy service, so the LB health check passes everywhere; the proxy forwards to web and preserves the client IP via the PROXY protocol or X-Forwarded-For with use-forwarded-headers: true.

Recommendation: C. It satisfies the client-IP requirement, keeps web at 3 replicas, makes the health check trivially true on all six nodes, gives you a single place to terminate TLS later, and - critically - it makes the existing NetworkPolicy correct instead of in the way (see next task). Option B is acceptable if adding a hop is politically unacceptable.

The policy does not match the traffic's actual source, and never will while the LB talks to the pods directly.

Read the policy carefully - it allows from: namespaceSelector: ingress-nginx on port: 8080. With a NodePort Service, the packet arrives at the pod from the external client's IP address (with Local, no SNAT occurs). That source is a CIDR, not a pod in the ingress-nginx namespace, so the namespaceSelector term never matches and the CNI drops the SYN. The pod's readiness probe may still pass, which makes it look like a random 60% failure rather than a policy drop.

Correct fix - allow the client CIDR:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-allow
  namespace: web
spec:
  podSelector:
    matchLabels: {app: web}
  policyTypes: [Ingress]
  ingress:
  - from:
    - ipBlock:
        cidr: 203.0.113.0/24          # the ranges the LB forwards from
    ports:
    - protocol: TCP
      port: 8080

Why not just change the port? Because port in a NetworkPolicy is the container port, not the Service port. The Service is port 80 -> targetPort 8080; the pod listens on 8080, so 8080 is already correct. Changing it to 80 would break the policy for legitimate traffic. This is the single most common misreading of NetworkPolicy and the reason the symptom looks like a port problem when it is a source problem.

If you chose Option C (ingress-nginx DaemonSet), the source is a pod in the ingress-nginx namespace, and the original policy becomes correct with no change at all - a good reason to prefer that design.

Debug recipe: kubectl get networkpolicy -n web -o yaml, confirm the pod's podIP and the client's observed source with kubectl exec <pod> -- ss -tn, and remember that a CNI that does not implement NetworkPolicy will silently allow everything - check the CNI before blaming the policy.