Cryptomining Detection in a Container Platform
A media company runs customer-facing workloads on Amazon EKS and AWS Lambda. The SOC sees a 300% increase in outbound traffic from one EKS namespace at 02:00, a GuardDuty finding for cryptocurrency DNS activity on an EC2 node, and an unexpected CloudWatch cost spike. The cluster uses Falco, but alerts go only to a Slack channel with no on-call rotation. EKS audit logs are enabled with seven-day retention. Developers share a cluster-admin role, and the same role can read Secrets Manager values. The incident-response plan covers on-premises servers but not containers or serverless. The SOC has one analyst on duty overnight. The security manager asks for immediate triage, containment, and a 30-day detection-engineering plan. The company is in the middle of a product launch and cannot take the entire cluster offline.
1. Triage the GuardDuty finding and traffic spike. What are your first 60 minutes, and how do you scope the compromise?
Read the worked answer
First confirm the finding is current, not stale, and correlate it with the traffic spike by namespace, pod, and node. Isolate the affected node and pod at the network layer with a quarantine security group or deny-all network policy, but preserve state before deletion. Snapshot the node's EBS volume, capture kubectl describe, pod logs, process lists, and container image digests. Rotate the node IAM role credentials and any Secrets Manager values readable by the shared cluster-admin role. Check CloudTrail for new access keys, role changes, and Secrets Manager reads; check EKS audit logs for exec, attach, and privileged pod creation.
Scope: all pods on the same node, all images from the same registry and build, and any Lambda functions using the same role or environment secrets. Look for persistence such as daemonsets, cronjobs, admission webhooks, and Lambda layers. Do not delete the workload until evidence is captured. The product launch constraint means isolate rather than shut down the cluster.
2. Contain and eradicate without taking the whole cluster offline. What order of operations?
Read the worked answer
Cordon and drain the affected node, then terminate it and rebuild from a clean AMI. Delete the malicious workload and any associated service accounts, roles, and secrets. Block egress to known mining pools at the network firewall and restrict the namespace's egress to approved destinations. Rotate all credentials that the pod or node could reach, including image registry credentials. Scan the remaining images for the same build artifacts and redeploy only verified images. Add an admission policy that blocks privileged pods, hostPath mounts, and hostNetwork unless explicitly approved.
Keep other namespaces running. Communicate to the product team which services are isolated and why. After eradication, monitor the namespace for re-infection and review whether the initial access vector was a vulnerable image, a leaked secret, or an overly permissive role. The order matters: isolate, preserve, rotate, rebuild, then restore.
3. Propose 30-day detection-engineering and logging improvements.
Read the worked answer
Within 30 days: route Falco, GuardDuty, EKS audit logs, VPC Flow Logs, and CloudTrail to the SIEM with an on-call escalation path, not a Slack channel. Extend EKS audit log retention to at least 90 days in immutable storage. Alert on privileged pod creation, exec into production pods, cluster-admin role changes, Secrets Manager bulk reads, and unusual egress volumes. Replace the shared cluster-admin role with scoped roles and a break-glass process. Add Kyverno or Gatekeeper policies to block privileged containers and require signed images. Deploy image scanning in CI and runtime scanning for running workloads. Write a container and serverless IR runbook and test it with a tabletop. Measure mean time to detect and mean time to isolate; review after the launch.