I85PREP
⌘ K
CKS study materials
CKS study resources

Three days later: no audit backend, a Falco DaemonSet nobody ships, and a compromised pod

CKS · Put your knowledge to work in context.

All case studies

Three days later: no audit backend, a Falco DaemonSet nobody ships, and a compromised pod

Five nodes, kube-apiserver running as a kubeadm static pod with no --audit-* flags set. Falco runs as a DaemonSet with the modern eBPF driver and its default output to stdout; there is no log shipper on the cluster. Namespace prod holds 60 pods, and the group developer is granted create on pods/exec through an aggregated ClusterRole. During an incident you learn that three days ago someone ran kubectl exec -it -n prod checkout-7c9f -- sh -c 'cat /etc/shadow'. You have no record of who, when, or from where. You are asked to make this class of activity detectable and to handle the next occurrence without destroying evidence.

1. Design the audit configuration to capture interactive access and sensitive reads, and state how it is applied and verified.

Read the worked answer

With no --audit-log-path and no webhook configured, the audit backend is disabled, so nothing was ever persisted - this is not a retention problem, it is an absent control. Note that the default policy (used when no policy file is supplied) logs at Metadata level, but only if a backend exists.

Minimal, cost-aware policy (audit.k8s.io/v1), rules evaluated in order:

rules:
- level: None
  nonResourceURLs: [/healthz, /livez, /readyz, /metrics]
- level: Request
  resources:
  - group: ""
    resources: [pods/exec, pods/attach, pods/portforward]
  verbs: [create]
- level: Metadata
  resources:
  - group: ""
    resources: [secrets, configmaps, serviceaccounts/token]
  verbs: [get, list, watch, create]
- level: Metadata
  omitStages: [RequestReceived]

exec is a create on the pods/exec subresource, and the command line appears in the event's requestURI query string, which is present at Metadata and above - so Request is enough and you never need RequestResponse for exec. Never log secrets at Request or RequestResponse: that writes secret bodies to disk in plaintext. omitStages: [RequestReceived] halves the volume.

Apply by adding --audit-policy-file=/etc/kubernetes/audit/policy.yaml, --audit-log-path=/var/log/kubernetes/audit/audit.log, --audit-log-maxsize, --audit-log-maxbackup=10, --audit-log-format=json to the kube-apiserver static pod, plus hostPath volumes and mounts for both the policy file and the log directory; the kubelet restarts the pod. Verify with kubectl exec -n prod <pod> -- true then grep pods/exec /var/log/kubernetes/audit/audit.log. Remember the log lives on the control-plane node's disk: it is lost with the node, so ship it off-box.

2. Falco has been running the whole time and did not alert. What are the plausible reasons, and how would you write and tune a rule for a shell in a container?

Read the worked answer

First, Falco alerts are not the same as Falco detecting. Output to stdout means the event went into the container's log stream, which is lost when the pod restarts and is invisible unless someone runs kubectl logs. A detection pipeline with no shipping is telemetry theatre: add a DaemonSet shipper (fluent-bit/vector) for /var/log/containers or a Falco sidecar, or set program_output/http_output to a collector. Second, verify the driver is actually instrumenting: kubectl logs -n falco ds/falco | grep -i 'driver\|eBPF' - modern-bpf silently degrades on some kernels and managed-node configurations. Third, check the rule fired but was suppressed by a default macro or an exception that is broader than intended.

A rule:

- rule: Interactive Shell in Application Container
  desc: A shell was spawned inside a workload container
  condition: >
    spawned_process and container
    and proc.name in (sh, bash, dash, ash, zsh)
    and not proc.pname in (runc, containerd-shim, crun)
  output: >
    Shell spawned in container (user=%user.name pod=%k8s.pod.name ns=%k8s.ns.name
    cmd=%proc.cmdline parent=%proc.pname image=%container.image.repository)
  priority: Notice
  tags: [container, shell]

Tune by namespace rather than by disabling: add and not k8s.ns.name in (kube-system) or a named exception block listing the specific workload images that legitimately start shells (init containers in some base images do). Keep the broad rule and narrow the exception list, so a new shell gets caught by default.

3. A `prod` pod is now suspected of being compromised. Give the containment and evidence steps in order, and say what you deliberately do not do.

Read the worked answer

Do not kubectl delete the pod first. Deleting it destroys the container filesystem and any process state, and the ReplicaSet immediately replaces it, so you lose both evidence and the artefact you were investigating.

  1. Isolate the network, not the pod. kubectl label pod <name> -n prod quarantine=true and apply a NetworkPolicy selecting quarantine=true with policyTypes: [Ingress, Egress] and no rules. Caveats: conntrack entries for existing connections may persist briefly, and a hostNetwork pod is unaffected - check spec.hostNetwork and, if set, cordon the node instead.
  2. Capture volatile state before anything restarts. kubectl get pod -o yaml and the ReplicaSet/Deployment spec (with the resolved image digest), kubectl logs --previous and current logs, kubectl describe pod for events, and kubectl get events -n prod.
  3. Inspect without trusting the container's own tools. Use kubectl debug -it <pod> --image=<hardened-tools> --target=<container> so the ephemeral container shares the target's process namespace; an attacker who replaced ps or netstat in the app image should not be the tool you investigate with.
  4. Scope the blast radius. Determine what the pod could reach: its ServiceAccount and kubectl auth can-i --list --as=system:serviceaccount:prod:<sa>, mounted secrets and ConfigMaps, and - most important - whether automountServiceAccountToken gave it API credentials. Treat every reachable secret as compromised.
  5. Remediate. Rotate or delete the ServiceAccount tokens and any credentials the pod could read, roll the workload from a known-good pinned digest, and only then remove the quarantined pod. Finally, check the audit log for pods/exec on that pod to reconstruct who was inside it, and review the developer group's pods/exec grant - exec into production is itself a privileged operation, not everyday access.