Rotated Secret Not Taking Effect, plus a Hardening Review

Secret as env var: never updates. envFrom is resolved by the kubelet exactly once, when it builds the container's process environment. The value is frozen for the life of that container; the only way to change it is to restart the container, which is what your colleague's pod deletion did. There is no propagation at all - "four hours" would have been "four months".

ConfigMap as a volume: updates, eventually, and only for the file - not the app's memory. The kubelet refreshes the projected contents on a sync loop; the observed latency is typically configurable around a minute but can be several minutes, and it is not bounded by anything you can set from the Pod. So the file on disk is probably new already - the app simply read /etc/app/config/app.yaml once at startup and never re-opened it. Verify with kubectl exec ... -- cat /etc/app/config/app.yaml: if it shows the new value, the volume is fine and the app is the problem.

The subPath exception is the one that actually bites. If the mount is mountPath: /etc/app/config/app.yaml, subPath: app.yaml, the kubelet never updates the file, because subPath uses a bind mount that is fixed at container start. The config secret/configmap will look stale forever with no error.

Two other traps: a ConfigMap or Secret marked immutable: true rejects updates outright (kubectl apply fails with a field-immutable error), and optional: true refs silently mount nothing rather than failing, so the app falls back to defaults.

Practical consequence: treat every env-sourced value as requiring a rollout. Either annotate the pod template with a checksum of the Secret/ConfigMap so kubectl apply rolls it, or accept kubectl rollout restart deploy/payments as part of the rotation runbook, and make sure the rotation procedure atomically updates the Secret for both old and new credentials so pods mid-rollout still authenticate.

spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000            # makes mounted volumes group-writable for UID 1000
    seccompProfile: {type: RuntimeDefault}
  volumes:
  - name: tmp
    emptyDir: {}
  - name: spool
    emptyDir: {}
  containers:
  - name: payments
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities: {drop: ["ALL"]}
    volumeMounts:
    - {name: tmp,   mountPath: /tmp}
    - {name: spool, mountPath: /var/spool/payments}

Every field here can fail in a specific way:

• runAsNonRoot: true with an image whose USER is root fails at container creation, not at runtime: CreateContainerConfigError / container has runAsNonRoot and image will run as root. runAsUser: 1000 overrides the image's user so it will start - but if the app binary or its directories are not readable by UID 1000, you now get permission denied in the logs instead. Check stat-able files: kubectl exec is unavailable for root-owned dirs, so test with a kubectl debug copy first.
• readOnlyRootFilesystem: true breaks the app unless both writable paths are mounted. /tmp is the one people remember; /var/spool/payments is the one that fails at 03:00. If the app also writes a PID file to /var/run, that needs a third mount.
• drop: ["ALL"] removes NET_BIND_SERVICE, so if the container listens on a port below 1024 it will fail with permission denied on bind. Check the containerPort and the app's bind address; remap to 8080+ or add back only NET_BIND_SERVICE.
• fsGroup: 1000 only takes effect for volumes that support ownership changes (emptyDir, PVC with the right driver). It does nothing for hostPath or some CSI-backed volumes, where ownership must be pre-set.
• allowPrivilegeEscalation: false interacts badly with setuid binaries in the image - if the app was silently relying on setuid to drop privileges, it now cannot.

Validate in a throwaway namespace before touching payments: kubectl run --dry-run=server catches only schema errors, not runtime permission failures, so apply to a canary Deployment with one replica.

kubectl get pod -l app=payments
# NAME                        READY   STATUS                       RESTARTS
# payments-6c9f7b8d4-2xk9p    0/1     CreateContainerConfigError   0

kubectl describe pod <pod> | tail -15
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].state.waiting.reason}: {.status.containerStatuses[0].state.waiting.message}{"\n"}'

CreateContainerConfigError means the kubelet could not build the container's config - the container has not started, so there are no logs to read. kubectl logs returns container ... is waiting to start, which sends people down the wrong path.

Three causes, in order of likelihood for this exact setup:

1. A referenced Secret/ConfigMap key does not exist. envFrom on a Secret, plus any env: valueFrom: secretKeyRef for a key that the rotation runbook forgot to recreate. Common when rotation is done as "delete and recreate" - the pods that tried to start in the gap are stuck permanently, because the kubelet does not retry a failed config resolution instantly and the Deployment has no reason to make a new pod. Fix the Secret, then kubectl rollout restart deploy/payments.
2. runAsNonRoot: true with an image whose USER is root - message: container has runAsNonRoot and image will run as root. This is exactly the state this image is in before you set runAsUser: 1000.
3. A missing volume/mount mismatch - a volumeMounts entry naming a volume that does not exist in spec.volumes; the API server rejects the Pod at admission, so you would see it as an event rather than this status.

The key judgement: this status is never an application or probe problem, and kubectl logs/kubectl exec are useless. The waiting.message on the container status is the only field that tells you which of the three it is - read it before editing anything, because the fix for (1) is a data change and the fix for (2) is a spec change.