A signature policy in Audit mode and a registry that lets tags be overwritten

Two independent failures. validationFailureAction: Audit means Kyverno records a PolicyReport and admits the pod anyway - a policy nobody reads is documentation, not a control. And mutateDigest: false means the admitted pod keeps its tag, so the pod spec does not record the identity that was actually verified.

Staged rollout, in this order:

1. Inventory the gap: kubectl get policyreport,clusterpolicyreport -A and the Kyverno metrics for fail results, so you know which workloads are currently unsigned.
2. Sign or pin the images that matter, or you will simply block your own deploys on the first enforced pod.
3. Flip to Enforce with mutateDigest: true and a failure mode chosen deliberately. Kyverno's default failurePolicy: Fail means a Kyverno outage blocks all pod creation in matched namespaces; run at least three replicas, tune webhookTimeoutSeconds, and accept the fail-closed behaviour consciously rather than discovering it.
4. Scope the policy by namespaceSelector - but verify Kyverno's default resourceFilters in its webhooks ConfigMap rather than assuming kube-system is excluded. Enforcing signature checks on the CNI or CoreDNS images in kube-system can make the cluster unrecoverable when the webhook is down.
5. Start with pod-security-like blast-radius thinking: enforce first on namespaces you deploy, keep an explicit, reviewed exception list, and watch the first 24 hours of Kyverno admission failures.

A tag is a mutable pointer; the digest is the identity. Cosign signs the digest (the manifest), and verification resolves tag to digest at verification time. Because this registry allows tag overwrite, a re-push between the moment CI signed 1.4.2 and the moment a node pulls it replaces the content the signature describes, and the pod that runs has a different digest than the one verified. imagePullPolicy: Always makes this worse rather than better: it guarantees the node re-resolves the tag to whatever is current, whereas IfNotPresent at least reuses the verified copy.

The fix in the manifests is a digest reference: registry.example.com/payments/api@sha256:..., and mutateDigest: true in the Kyverno policy so any tag-based manifest is rewritten to the digests that were verified before the pod object is persisted. Workflows:

• get the digest deliberately in CI: crane digest registry.example.com/payments/api:1.4.2 or skopeo inspect --format '{{.Digest}}' docker://...;
• sign that digest, not the tag, and verify with cosign verify --key cosign.pub registry.example.com/payments/api@sha256:...;
• after deploy, confirm what is actually running: kubectl get pods -n payments -o jsonpath='{.items[*].status.containerStatuses[*].imageID}' should show the same digest you signed;
• additionally enable tag immutability in the registry if it supports it, so a compromised CI credential cannot retag at all.

Never express the exception as a namespace label matched by a global namespaceSelector whose label key is documented, because anyone who can patch namespaces inherits the bypass to a signed-images-only cluster. Use enforcement-aware scoping:

• Kyverno PolicyException (or the equivalent per-policy exclude list) scoped to debug, owned by the on-call team, reviewed and time-boxed, and created by a Role that only the on-call group holds. The exception is a reviewable object with an audit trail, unlike a label edit.
• Make the namespace inherently low-value: pod-security.kubernetes.io/enforce: privileged with RBAC that permits only the on-call group to create pods there; a default-deny NetworkPolicy so a hostile image cannot reach the API server or other workloads; automountServiceAccountToken: false on the namespace's ServiceAccounts; no mounted secrets from payments; and a ResourceQuota so it cannot be used to mine crypto at cluster scale.
• Do not let it escalate: deny hostPath mounts of /var/run/containerd.sock and /etc/kubernetes via a separate policy, since those turn a debug pod into node root regardless of image signing.
• Audit it as a first-class exception: alert on any pod created in debug (Falco or an audit rule), and on any creation of new PolicyException objects. The point of the exception is that it is small, loud and attributable - not that it is secret.