Crypto Mining on 14 Nodes After a Leaked CI Token
At 02:14, monitoring alerts that CPU on 14 of 80 nodes is above 95%. Investigators find pod web-cache-7f9 in namespace frontend running an xmrig binary. The pod image is nginx:1.25 from a public registry, but the running container's image digest differs from the deployment spec. The pod has no CPU limits, mounts hostPath /var/log, and uses a service account with get and list on Secrets in its namespace. Logs show the pod was created by a token belonging to ci-runner, whose token was committed to a public Git repository three weeks earlier. The cluster has audit logging to a local file, no admission policy for image signatures, and network policy only in two namespaces. The incident commander must decide containment, eradication, and evidence preservation while production checkout stays online.
1. Reconstruct the most likely attack path from the public Git repository to cryptomining, naming the evidence that supports each step.
Read the worked answer
The chain likely starts with the ci-runner token committed to public Git. An attacker uses it to authenticate to the API and create or modify a pod in frontend, possibly by patching the existing Deployment. The running digest differs from nginx:1.25 because the tag was overwritten or the pod spec pulled a different image; no signature admission allowed it.
The pod has no CPU limits, so xmrig consumes node CPU and triggers the alert. hostPath /var/log gives log access; the namespace service account can list Secrets for further credential theft. The public registry and lack of image verification explain initial code execution; the long-lived token explains API access.
The 14 nodes at 95% suggest either a DaemonSet-like spread or repeated pod creation across nodes, and audit logs on local disk may be incomplete. Each step has a matching evidence source: Git history, API audit events, image digest mismatch, and CPU metrics.
2. Which missing controls or detection gaps would have broken the chain earliest, and why?
Read the worked answer
The earliest break would be secret scanning and revocation of the committed token, because it is the initial access. Next, admission control requiring signed images and immutable digests would stop the running container even with a valid deployment. Third, disabling automount and scoping the ci-runner service account would limit what the token can do. Fourth, CPU limits and runtime detection for mining process names would shorten dwell time. Fifth, default-deny network policy would limit secret access and command-and-control. Centralized audit logs would make the pod creation visible.
The reasoning is to break the chain at initial access and execution, not only detect the final CPU symptom. A control that only alerts on high CPU still allows credential theft and persistence before response.
3. As incident commander, what are your first containment and evidence-preservation actions, and what would you avoid doing?
Read the worked answer
Contain: revoke and rotate the ci-runner token, then disable the account; cordon the 14 nodes and stop new scheduling; apply a quarantine network policy or label to the frontend namespace; preserve the pod, node, and audit logs by snapshotting before deletion. Preserve local audit files because they may be the only record.
Eradicate: rebuild affected nodes from a clean image rather than cleaning in place, rotate secrets the pod could read, and audit CI logs and Git history.
Avoid immediately deleting the pod or wiping nodes, because that destroys evidence and may hide persistence. Keep checkout online by shifting traffic to clean nodes after verifying no other compromised pods. The reasoning is to stop spread first, then preserve evidence, then rebuild with rotated credentials.