Responding to a Privileged Container Escape in a Shared Cluster
A multi-tenant Kubernetes platform on GKE hosts workloads for 12 business units, including a PCI-regulated payment team. The cluster has 300 pods across 20 namespaces. Falcon Cloud Security detects a pod in the 'dev-sandbox' namespace running a privileged container that attempts to mount the host filesystem. The pod was created by a developer with cluster-admin permissions. The detection fires at 10:00 on a weekday. The platform team must respond without disrupting production workloads in other namespaces. The PCI team requires that any incident touching the cluster be documented and that containment not violate change management. The security team has Falcon RTR for Kubernetes and can use kubectl. They need to decide whether to delete the pod, cordon the node, or isolate the namespace, and how to preserve evidence.
1. Evaluate the containment options: deleting the pod, cordoning the node, or isolating the namespace. Which do you choose and why?
Read the worked answer
Deleting the pod is fast and removes the immediate threat, but it destroys container-local evidence and may not stop a pod recreated by a controller. It also does not address whether the host filesystem was actually mounted.
Cordoning the node prevents new pods from scheduling but leaves existing pods running, including the malicious one. It is too broad if the node hosts production workloads and too weak if the attacker already has host access.
Isolating the namespace with a deny-all NetworkPolicy limits lateral movement but may be bypassed by a cluster-admin and does not stop a host mount already in progress.
The best sequence is: use Falcon RTR and kubectl to capture pod details, logs, and node state; apply a deny-all NetworkPolicy to dev-sandbox; delete the malicious pod; then cordon and drain the node if there is any sign the host filesystem was mounted. This contains the incident without disrupting production namespaces and preserves evidence before deletion. Because the namespace is a dev sandbox, the business impact of isolation is low, which makes it a reasonable first containment step.
2. How would you preserve forensic evidence while meeting the PCI documentation requirement?
Read the worked answer
Before deleting the pod, run kubectl describe pod, kubectl logs, and kubectl get pod -o yaml to capture the pod spec, service account, and volume mounts. Use Falcon RTR to collect the process tree, open files, network connections, and container image hash. If the host filesystem was mounted, snapshot the node's boot disk and preserve the container runtime logs.
Record every action in the incident ticket with timestamps, operator name, and justification. Store the collected artifacts in a write-once or versioned bucket with restricted access. For PCI, map the documentation to the incident response control: detection time, containment time, evidence collected, and change approval. Because the containment action may be an emergency change, attach the emergency change record and the PCI team's approval. The goal is a chain of custody that shows the investigation was deliberate and the production environment was protected.
3. What access control changes would you recommend to prevent recurrence?
Read the worked answer
Remove cluster-admin from developers. Implement role-based access control with least privilege, and separate duties so developers can deploy to their namespaces but cannot create privileged pods or modify cluster-wide resources. Use Kubernetes admission control, such as Pod Security Standards or a policy engine, to block privileged containers, hostPath mounts, and host network access by default.
Enforce namespace isolation with NetworkPolicies and resource quotas. Audit all cluster-admin bindings and service accounts, and rotate credentials. Enable Falcon Kubernetes admission control so detections can prevent risky workloads before they run. For the PCI namespace, apply stricter controls and monitoring, and require approval for any workload that requests elevated privileges. Finally, provide developers with a safe sandbox that cannot reach the host or other namespaces, so experimentation does not require cluster-admin rights.