Empty Endpoints, a Default-Deny Policy, and Connection Refused
Namespace api hosts the orders Deployment (pod template labels app=orders, tier=back-end) behind Service orders-svc (port 80 to targetPort 8080, selector app=orders, tier=backend). Namespace web runs the frontend (app=web). A new NetworkPolicy in api selects all pods (podSelector: {}, policyTypes Ingress and Egress) with no rules. After it was applied, kubectl get endpoints orders-svc returns <none>, frontend calls fail, and the orders pods log no incoming requests. The orders container defines no readiness probe; its pods are Running. A second namespace batch (label team=batch) must also reach orders on 8080, and orders needs cluster DNS plus an external payment API on 443.
1. Is the NetworkPolicy responsible for empty Endpoints? Diagnose the actual cause.
Read the worked answer
No. NetworkPolicy is enforced by the CNI plugin on packets. It does not touch EndpointSlices: those are computed by the endpoint controller from (a) the Service selector, (b) each Pod's labels, and (c) each Pod's Ready condition. A default-deny policy can make traffic fail, but it can never make kubectl get endpoints print <none>.
That means there are two independent faults here - the empty Endpoints and the blocked traffic - and fixing them out of order wastes time.
kubectl -n api get pods --show-labels | grep orders # app=orders,tier=back-end
kubectl -n api get svc orders-svc -o jsonpath='{.spec.selector}{"\n"}' # app=orders,tier=backend
Pull both sides of the selector into one view: the Service asks for tier=backend, the pods carry tier=back-end. Endpoint controller matches on all selector keys, so zero pods qualify and the EndpointSlice stays empty. The CNI is then handed nothing to forward to, which is why the frontend sees a failure and orders logs nothing at all - the packets never reach the namespace.
Note the probe detail: with no readiness probe, a Running pod is Ready by default, so readiness is not a factor here. But once you fix the label, add a readiness probe anyway - the moment you add one, a broken readiness probe produces exactly the same empty-Endpoints symptom, and you will want that diagnosis to be a one-liner next time.
Fix: either patch the Service (kubectl -n api patch svc orders-svc -p '{"spec":{"selector":{"app":"orders","tier":"back-end"}}}') or relabel the Deployment template and roll. Prefer fixing the Service if other clients depend on the pod label as-is, and prefer relabelling if tier=backend is the documented convention - decide by which one other Deployments and policies already assume.
2. Write the policy that lets exactly `web` and `batch` reach orders on 8080, while orders keeps DNS and its egress to the payment API.
Read the worked answer
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: orders-allow, namespace: api}
spec:
podSelector:
matchLabels: {app: orders}
policyTypes: [Ingress, Egress]
ingress:
- from:
- namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: web}}
podSelector: {matchLabels: {app: web}}
ports: [{protocol: TCP, port: 8080}]
- from:
- namespaceSelector: {matchLabels: {team: batch}}
ports: [{protocol: TCP, port: 8080}]
egress:
- to:
- namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}
ports:
- {protocol: UDP, port: 53}
- {protocol: TCP, port: 53}
- to:
- ipBlock:
cidr: 203.0.113.0/24
except: [203.0.113.128/25]
ports: [{protocol: TCP, port: 443}]
Four judgement points that make or break this:
- Two
fromentries, not one with two selectors.namespaceSelectorandpodSelectorin the samefromelement AND together ("web pods that are also web-namespace pods"); separate list items OR. Puttingteam=batchin the same element asapp=webwould allow nothing. kubernetes.io/metadata.nameis applied automatically to every namespace by the API server (since 1.21), sowebneeds no manual labelling.batchuses a customteam=batchlabel - confirm it exists, because an unlabelled namespace silently matches nothing and the failure looks identical to a policy bug.- The
podSelector: {}in the original policy is why nothing worked at all, and this replacement narrows the target toapp: orders- so other pods inapireturn to their previous (unrestricted) behaviour. Decide deliberately whether that is acceptable; if not, keep a separate default-deny policy and add explicit allow policies per workload. - Port 8080, not 80. Policy is evaluated after Service DNAT, so the destination port seen by the CNI is the pod's
targetPort. Writingport: 80produces a policy that drops every packet - and it looks correct in review.
3. After the label fix and the policy are in place, the frontend still gets `connection refused`. What does that error specifically tell you, and what are the candidate causes?
Read the worked answer
connection refused is a strong signal that the packet was delivered and rejected by the destination host - a TCP RST. A NetworkPolicy drop (and an empty Endpoints Service, which blackholes to a non-existent ClusterIP) manifests as a timeout, not a refusal. So connection refused means the policy and the Service plumbing are now working, and the problem is at the application/socket layer.
Candidates, in order:
targetPortmismatch. Service says 8080; the container actually listens on 8081 and nothing binds 8080. Verify from inside the cluster, not from the spec:kubectl -n api exec deploy/orders -- sh -c 'netstat -lntp || ss -lntp'- The app binds
127.0.0.1instead of0.0.0.0. The socket exists on loopback only, so the pod IP's 8080 is closed - refused, not timed out. Very common with dev-default configs. - The container's
containerPortis decorative. It is metadata only; changing it fixes nothing. OnlytargetPortand the actual bind address matter. - A kube-proxy/Service with no Endpoints after a partial fix would give a timeout again; if you see a timeout coming back, re-run
kubectl get endpointslice -n api -o wideand re-check readiness.
Diagnostic that separates 1/2 from policy immediately:
kubectl -n web run t --rm -it --image=nicolaka/netshoot -- \
curl -sv --max-time 5 http://orders-svc.api.svc.cluster.local/
Connection refused to the pod IP but success to 127.0.0.1 on the pod confirms a bind-address issue; timeout sends you back to the policy; no such host sends you to the DNS egress rule - which is the third distinct failure mode, and with a default-deny egress policy in web you will hit it before anything else.