Flipping a namespace to restricted: who breaks, and what you refuse to do about it

hostNetwork, hostPID and hostPath volumes are all violations of baseline, not just restricted, so node-exporter was already running outside the enforced policy - it only survived because PSA is evaluated at admission and the DaemonSet's pods predate the label change or were admitted while the label was looser. Flipping to restricted simply removes the illusion.

The right move is a separate namespace, e.g. node-observability, labelled pod-security.kubernetes.io/enforce: privileged (or baseline if only hostPath is needed and hostNetwork can be dropped), with three controls that make the exception defensible:

• RBAC allowing only the platform ServiceAccount to create workloads there - a privileged namespace is an escape hatch for anyone who can schedule into it;
• a default-deny NetworkPolicy, since a hostNetwork pod bypasses pod-network policies anyway, so also restrict what the node itself can reach;
• the hardened parts you can keep: seccompProfile: RuntimeDefault, read-only hostPath mounts (readOnly: true), runAsNonRoot if the exporter allows it, and a pinned, scanned image.

Do not relax legacy, and do not create a label such as psa: privileged that a namespaceSelector in a webhook trusts - that just moves the bypass to whoever can create namespaces. Namespace creation and namespace labelling are privileged operations; make sure they are not in the default developer ClusterRole.

Failure one is admission: under restricted, a pod without runAsNonRoot: true, without allowPrivilegeEscalation: false, without capabilities.drop: [ALL] and without an explicit seccompProfile is rejected outright. Failure two is runtime: patching only runAsNonRoot: true pushes the check down to the kubelet, which refuses to start a container whose image resolves to UID 0 - hence CreateContainerConfigError, not an admission error.

Pod-level context:

    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 101
        runAsGroup: 101
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: nginx
        image: registry.example.com/nginx-unprivileged@sha256:<digest>
        ports:
        - containerPort: 8080
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop: [ALL]

The image change is the real fix. The official nginx image needs root to bind :80 and to write its pid and cache directories; nginx-unprivileged listens on 8080 as UID 101. Update the Service targetPort accordingly and add emptyDir mounts for /var/cache/nginx and /tmp so readOnlyRootFilesystem: true does not crash the worker. Do not "fix" this by adding NET_BIND_SERVICE and running as root - restricted permits adding only NET_BIND_SERVICE to the drop list, but runAsNonRoot still applies, and a root nginx with a writable filesystem is exactly the workload the policy exists to prevent.

Not broken. Pod Security Admission is an admission-time control: the label affects only pod creation (and updates that recreate pods). Running pods are not evicted, which is why a --overwrite label change looks like a no-op in kubectl get pods. You must roll the workloads - kubectl rollout restart deploy/nginx-front -n legacy and delete DaemonSet/Job pods - and confirm the replacements are admitted. Anything that will not restart is your real violation inventory.

For report-gen, two things to check. First, restricted does not forbid shells; /bin/sh inside a container is fine and is a runtime-detection problem, not a PSA problem, so do not try to solve it with pod security. Second, Jobs and CronJobs are immutable in their pod template: editing the CronJob's jobTemplate does not change already-created Jobs, and a rejected Job leaves a FailedCreate event on the CronJob rather than an obvious error. Check kubectl describe cronjob report-gen and kubectl get events --field-selector reason=FailedCreate.

Finally, verify enforcement actually applies to the workload rather than assuming: kubectl auth can-i create pods -n legacy --as=system:serviceaccount:ci:deployer plus a dry-run - kubectl run t --image=nginx --dry-run=server -n legacy should be rejected with the PSA message naming the level.