Kyverno in Audit Mode, an Unpatchable Vendor Image, and 61 Critical CVEs
Kestrel Logistics runs 96 services on a shared EKS 1.28 cluster: 340 pods, 55 nodes, three node groups, and a legacy vendor image for yard scanning that ships only as a prebuilt tarball on a 14-month release cadence. A registry scan reports 61 images with critical CVEs; 22 of them are latest-tagged internal builds on the end-of-life node:14 base. A Kyverno policy set sits in Audit mode, flagging pods running as root, hostPath mounts, and images without a digest. Two weeks ago a compromised service account token was used to deploy a pod with a hostPath mount to /var/lib/kubelet, and nobody was paged because the violation only appeared in a daily report. The platform team wants to flip policies to Enforce on Monday; application teams say that will block the next release.
1. How do you move from Audit to Enforce without a production outage? Give the sequencing and say how exemptions should work.
Read the worked answer
Do not flip all three rules at once on Monday. Sequence by blast radius and fixability, and use data from Audit rather than opinion.
- Quantify first. Turn the existing audit results into a per-namespace, per-owner violation inventory and publish it. The argument about breaking releases collapses when each team sees its own 4 workloads rather than an abstract figure.
- Enforce the rules nobody legitimately needs.
hostPath,privileged, andhostNetworkcome first: workloads almost never need them, and this is the exact path the attacker used. One allowlist exception for the CNI and CSI DaemonSets inkube-system, expressed as a tailored policy for system namespaces rather than a blanket namespace exclusion. - Then digests, then
runAsNonRoot. Digest pinning is cheap to adopt. Non-root requires application changes and is where the schedule genuinely slips - give it the longest runway and enforce it in dev and staging first, then a wave-based production rollout. - Exemptions as named objects with an owner and an expiry. Use Kyverno
PolicyExceptionresources per workload, with labels for owner and expiry date, tracked in a report. Never exempt a whole namespace to unblock one pod. Anything that reaches its expiry without a fix becomes a failing check in CI. - Make the webhook safe. Run Kyverno with multiple replicas and set
failurePolicydeliberately; a failing webhook inFailmode can stop all pod creation in the cluster. Excludekube-systemand the CNI specifically, or an admission outage becomes a node outage.
Close the detection gap in the same change: route violation events to the SIEM with an on-call route, and add runtime detection. Admission stops the next bad deployment; it does not tell you about a pod already running as root.
2. The vendor image cannot be rebuilt. What is your risk treatment?
Read the worked answer
Treat it as an accepted, contained risk, not an exception to the policies.
- Contain the blast radius. Run it in a dedicated node group with taints and tolerations so no other workload shares the node; a dedicated namespace; a NetworkPolicy that allows only the required egress to the yard-scanning endpoints and DNS, with everything else denied.
- Remove its privileges. No
hostPath, noprivileged, nohostNetwork,readOnlyRootFilesystemwhere the vendor will tolerate it,allowPrivilegeEscalation: false, a read-only seccomp profile, andautomountServiceAccountToken: falseunless it genuinely calls the API - and then a dedicated service account with a minimal role and bound tokens. - Pin and mirror. Import the vendor tarball into ECR, reference it by digest, and sign it so a tag substitution cannot swap the image. Record the vendor's SBOM if one is supplied; if not, ask, and record the refusal.
- Patch cadence as a contractual issue. The 14-month release cycle is the real finding. Put it on the vendor risk register with the next release date, and get the vendor to confirm whether the image is still supported. Procurement leverage belongs in the risk acceptance, not in a ticket.
- Time-boxed exception. Owner, justification, compensating controls listed, expiry aligned to the vendor's next release, and a review at renewal. Report it to the risk committee and re-test the compensating controls quarterly.
What you must not do is exempt the namespace and let it drift into an unmanaged shadow of the cluster. The exception should make the workload easier to audit, not less visible.
3. How do you prioritise the 61 critical CVEs, and where does the hostPath incident fit?
Read the worked answer
The incident tells you the CVEs are not the top risk. The compromise was not a CVE: it was a stolen token plus an over-privileged pod spec. Fixing admission and token scope reduces likelihood more than clearing 61 scanner rows. Say so explicitly, because the instinct will be to run a CVE burn-down and skip the privilege work.
Prioritisation inputs, in order:
- Exploitability: is the CVE in CISA KEV, what is the EPSS score, and is a public exploit or a working PoC available? A critical CVSS with no exploit path ranks below a high CVSS that is being actively exploited.
- Exposure: is the service reachable from the internet through the ingress, or only from inside the mesh? Internet-facing plus exploitable jumps the queue.
- Reachability: is the vulnerable library actually on the request path at runtime, or a build-time dependency that never executes? Runtime evidence beats scanner heuristics.
- Fix availability: an upgrade that exists beats a mitigation you have to write.
Programme-level findings, handled separately from per-image tickets:
- The 22
node:14builds are an end-of-life base image problem. Track it as one migration to a supported base with an owner and date, and block new builds on the EOL base in CI rather than opening 22 tickets. latesttags defeat reproducibility and make CVE attribution unreliable. Fix digests and automated rebuild-on-base-update so images stop accumulating months of drift silently.
Report: two numbers, not one - count of exploitable, internet-reachable criticals over time (should go to zero) and count of total criticals (will move slowly and is not the metric the board should watch).