EKS runtime alert storm, one real intrusion, and blind nodes

Work the payments alert first, despite it being one alert against 340. The decision is about behavior and blast radius, not count: a shell spawned network tooling from a long-running service process, and the pod identity can read the tokenization bucket. The ETL alerts are process drift in a batch workload; the payments alert is a plausible data-theft chain.

Evidence before containment, because deleting the pod destroys the process tree:

• Full process tree with user, command line and binary hash; whether curl wrote a file that was then executed.
• The deployed image digest versus the scanned digest, to rule out a swapped image.
• Pod service account, IRSA role, and CloudTrail for s3 access by that role in the last 24 hours.
• VPC flow or DNS logs for the destination IP; whether egress from that namespace is normally permitted.
• Whether other pods on the same node or same image show the same process sequence.

Then contain: network policy denying egress for the namespace, cordon the node, snapshot and scale the deployment down, and rotate the pod credentials since the session may already be used. Do not flip the runtime policy to block mode mid-incident; scope and test that change afterwards.

This is a policy scoping problem, not a policy problem. The batch job's entrypoint is expected, so the rule should not fire for it - but the fix must not cover namespace etl with a wildcard on /bin/sh, because that would blind the exact detection that mattered in payments.

Ranked options:

1. A runtime rule scoped to namespace etl plus the specific image digests, allowing the exact process path of the wrapper while leaving every other process alerting. Because it is pinned to digests, a rebuilt image re-alerts until reviewed - which is the behavior you want.
2. If the wrapper path changes per release, allow the parent binary only and keep child-process detection active.
3. If neither can be expressed, downgrade the rule to alert-only for that namespace with a named owner and an expiry date, rather than disabling it globally.

Record each tuning as an exception with owner, justification, and a review date, and require security review for any change that expands scope beyond one namespace. Then measure: alert volume from etl should drop to near zero while a synthetic test process in a canary namespace still fires. Tuning that nobody re-tests is just slow deletion of coverage.

A 256Mi limit is too low for a node hosting many containers once the agent does image and process work; the kernel kills it and the node goes dark. Raise the limit to a realistic value (512Mi to 1Gi depending on pod density) with a matching request so the pod gets guaranteed QoS rather than being first in the eviction order, and consider a high-priority class. Do not simply remove the limit - that risks node-level memory pressure and takes down customer workloads.

Verify the fix per node, not per cluster: check agent health in the console and alert when an agent has not reported for more than 5 minutes. That alert is what was missing this time.

Coverage during the gap needs to be stated precisely rather than waved away:

• Cloud posture assessment (CSPM) is API-based and unaffected, so config drift on those nodes' workloads is still visible.
• Admission control runs as a separate webhook rather than inside the DaemonSet, so blocking may still work; confirm it is healthy on cluster-b before assuming protection.
• What is genuinely lost is runtime process, network and file behavior on those nodes. Compensate with network egress restriction on sensitive namespaces, and state the residual risk explicitly in the incident record.