CCSO study materials
CCSO study resources

Kyverno in Audit Mode, an Unpatchable Vendor Image, and 61 Critical CVEs

CCSO · Put your knowledge to work in context.

All case studies

Kyverno in Audit Mode, an Unpatchable Vendor Image, and 61 Critical CVEs

Kestrel Logistics runs 96 services on a shared EKS 1.28 cluster: 340 pods, 55 nodes, three node groups, and a legacy vendor image for yard scanning that ships only as a prebuilt tarball on a 14-month release cadence. A registry scan reports 61 images with critical CVEs; 22 of them are latest-tagged internal builds on the end-of-life node:14 base. A Kyverno policy set sits in Audit mode, flagging pods running as root, hostPath mounts, and images without a digest. Two weeks ago a compromised service account token was used to deploy a pod with a hostPath mount to /var/lib/kubelet, and nobody was paged because the violation only appeared in a daily report. The platform team wants to flip policies to Enforce on Monday; application teams say that will block the next release.

1. How do you move from Audit to Enforce without a production outage? Give the sequencing and say how exemptions should work.

Read the worked answer

Do not flip all three rules at once on Monday. Sequence by blast radius and fixability, and use data from Audit rather than opinion.

  1. Quantify first. Turn the existing audit results into a per-namespace, per-owner violation inventory and publish it. The argument about breaking releases collapses when each team sees its own 4 workloads rather than an abstract figure.
  2. Enforce the rules nobody legitimately needs. hostPath, privileged, and hostNetwork come first: workloads almost never need them, and this is the exact path the attacker used. One allowlist exception for the CNI and CSI DaemonSets in kube-system, expressed as a tailored policy for system namespaces rather than a blanket namespace exclusion.
  3. Then digests, then runAsNonRoot. Digest pinning is cheap to adopt. Non-root requires application changes and is where the schedule genuinely slips - give it the longest runway and enforce it in dev and staging first, then a wave-based production rollout.
  4. Exemptions as named objects with an owner and an expiry. Use Kyverno PolicyException resources per workload, with labels for owner and expiry date, tracked in a report. Never exempt a whole namespace to unblock one pod. Anything that reaches its expiry without a fix becomes a failing check in CI.
  5. Make the webhook safe. Run Kyverno with multiple replicas and set failurePolicy deliberately; a failing webhook in Fail mode can stop all pod creation in the cluster. Exclude kube-system and the CNI specifically, or an admission outage becomes a node outage.

Close the detection gap in the same change: route violation events to the SIEM with an on-call route, and add runtime detection. Admission stops the next bad deployment; it does not tell you about a pod already running as root.

2. The vendor image cannot be rebuilt. What is your risk treatment?

Read the worked answer

Treat it as an accepted, contained risk, not an exception to the policies.

  • Contain the blast radius. Run it in a dedicated node group with taints and tolerations so no other workload shares the node; a dedicated namespace; a NetworkPolicy that allows only the required egress to the yard-scanning endpoints and DNS, with everything else denied.
  • Remove its privileges. No hostPath, no privileged, no hostNetwork, readOnlyRootFilesystem where the vendor will tolerate it, allowPrivilegeEscalation: false, a read-only seccomp profile, and automountServiceAccountToken: false unless it genuinely calls the API - and then a dedicated service account with a minimal role and bound tokens.
  • Pin and mirror. Import the vendor tarball into ECR, reference it by digest, and sign it so a tag substitution cannot swap the image. Record the vendor's SBOM if one is supplied; if not, ask, and record the refusal.
  • Patch cadence as a contractual issue. The 14-month release cycle is the real finding. Put it on the vendor risk register with the next release date, and get the vendor to confirm whether the image is still supported. Procurement leverage belongs in the risk acceptance, not in a ticket.
  • Time-boxed exception. Owner, justification, compensating controls listed, expiry aligned to the vendor's next release, and a review at renewal. Report it to the risk committee and re-test the compensating controls quarterly.

What you must not do is exempt the namespace and let it drift into an unmanaged shadow of the cluster. The exception should make the workload easier to audit, not less visible.

3. How do you prioritise the 61 critical CVEs, and where does the hostPath incident fit?

Read the worked answer

The incident tells you the CVEs are not the top risk. The compromise was not a CVE: it was a stolen token plus an over-privileged pod spec. Fixing admission and token scope reduces likelihood more than clearing 61 scanner rows. Say so explicitly, because the instinct will be to run a CVE burn-down and skip the privilege work.

Prioritisation inputs, in order:

  • Exploitability: is the CVE in CISA KEV, what is the EPSS score, and is a public exploit or a working PoC available? A critical CVSS with no exploit path ranks below a high CVSS that is being actively exploited.
  • Exposure: is the service reachable from the internet through the ingress, or only from inside the mesh? Internet-facing plus exploitable jumps the queue.
  • Reachability: is the vulnerable library actually on the request path at runtime, or a build-time dependency that never executes? Runtime evidence beats scanner heuristics.
  • Fix availability: an upgrade that exists beats a mitigation you have to write.

Programme-level findings, handled separately from per-image tickets:

  • The 22 node:14 builds are an end-of-life base image problem. Track it as one migration to a supported base with an owner and date, and block new builds on the EOL base in CI rather than opening 22 tickets.
  • latest tags defeat reproducibility and make CVE attribution unreliable. Fix digests and automated rebuild-on-base-update so images stop accumulating months of drift silently.

Report: two numbers, not one - count of exploitable, internet-reachable criticals over time (should go to zero) and count of total criticals (will move slowly and is not the metric the board should watch).