Nine days of cryptomining behind 1,900 alerts a day
A three-analyst SOC monitors five account groups in Prisma Cloud. Every High and Critical alert is routed to a single Slack channel and a shared mailbox: about 1,900 alerts a day, most of them repeat configuration findings on the same 40 long-lived resources. Last month a cryptomining workload ran on an EC2 instance in the payments account for nine days; it was found by the cloud cost report, not by the SOC. A behavioural alert on that instance had arrived in the channel overnight and was never assigned. The same period produced roughly 400 container runtime alerts per week from a "process spawned in container" rule, almost all from three namespaces that run CI jobs. The CISO's requirements are: no genuinely critical event may sit unacknowledged for more than one hour, every high-risk alert gets a 24-hour acknowledgement, and you must be able to prove the pipeline works.
1. Redesign alert routing and triage. State which alerts go to which destination, what filtering mechanism you use, and the reasoning behind each choice.
Read the worked answer
Principle: route by the action required, and use severity to set the SLA, not to pick the channel. Today one channel carries everything, so urgency is invisible.
Tier 1 - page a human now (on-call/PagerDuty or equivalent). Runtime and behavioural detections on production: crypto-mining patterns, reverse shells, unusual outbound connections, privilege escalation on a host, container escape indicators; plus IAM/credential exposure and public data stores containing tagged sensitive data. These are the alerts where minutes matter. Build this as one alert rule filtered on policy label/type = runtime-behavioural plus severity, scoped to production account groups. Labels are the mechanism that lets you say "this class of alert is tier 1" without maintaining a list of policy names.
Tier 2 - create a ticket (ServiceNow/Jira) with a 24-hour SLA. High-severity config findings on internet-exposed production resources: open security groups to sensitive ports, public buckets with effective exposure, missing encryption on regulated data. Ticketing gives you an owner, a timestamp, and evidence of acknowledgement - which the shared mailbox does not.
Tier 3 - digest, no per-alert notification. Everything else, including repeat findings on the same 40 resources. Collapse by resource and policy so the digest shows counts, not 1,900 rows.
Kill the specific noise at source. The 400 weekly "process spawned" alerts are almost certainly CI jobs behaving normally. Scope that runtime rule to exclude the CI namespaces (namespace/image/label scoping), and only alert on processes deviating from the learned baseline. Do not simply raise the threshold until it is silent - you would miss the one real spawn.
Deduplicate deliberately. One alert per resource per policy, then aggregate in the destination by resource ID and policy, and use saved searches to view open findings rather than re-reading notifications.
2. Reconstruct the missed cryptomining incident: what would you pull from Prisma Cloud, in what order, and what would you have needed in place to catch it on day one?
Read the worked answer
Day-one detection requires the alert to be assigned, so the first gap is process, not data. Reconstruct as follows.
- The alert itself. Resource ID, account, region, policy, first-seen and last-seen, and the full detail payload. "Last seen" tells you whether it is ongoing; a nine-day-old alert still firing means the workload was never contained.
- Compute-side view. If a Defender covered the host, go to Incidents/Forensics for that workload: the process tree that triggered the rule, its parent, the binary path, outbound connections and DNS lookups, and any file-system changes. This is what distinguishes a miner from a legitimate batch job.
- Cloud-side configuration via RQL. Query the instance: security groups and whether anything is exposed, public IP, attached instance profile, user data, and the AMI. Then query for other instances launched from the same AMI or with the same anomalous pattern - miners spread.
- Blast radius. Enumerate what the instance profile could reach (S3 buckets, secrets, other accounts' roles) and check whether that IAM role was used from anywhere unexpected. Rotate the credentials regardless; assume they were readable from the host.
- Containment and evidence. Snapshot the volume and capture logs before terminating, isolate the instance at the network layer, then rebuild from a known-good image.
What had to exist on day one: a tier-1 route for behavioural detections that pages rather than posts, a named owner per shift, and a runbook that says "behavioural alert on a production host = open an incident, do not triage from the Slack preview". The data was there; the routing and ownership were not. Also verify that the host actually had runtime coverage - a nine-day blind spot is a coverage failure if no Defender was deployed.
3. How do you prove to the CISO that the new pipeline works, and which metrics would you put on a monthly report?
Read the worked answer
Prove it with controlled tests, not with volume. Volume is exactly the metric that was misleading before.
Canary detections. Once a month, in a designated test account: create a security group open to 0.0.0.0/0 on 22, create a bucket with an effectively public policy, and run a benign process that triggers the behavioural rule. Measure detection time and acknowledgement time end to end. Record the expected alert ID. This is the only credible evidence that a rule change has not silently broken a route.
Coverage checks. For every policy in the tier-1 and tier-2 sets, confirm it is enabled, mapped to an alert rule, and that the destination integration is healthy. A policy with no alert rule is worse than a disabled one because it looks active.
Metrics for the monthly report:
- Alert-to-incident ratio (how many notifications produced an investigation, and how many investigations found something real).
- Median time to acknowledge, split by tier - report the tier-1 median against the one-hour target.
- Count of open tier-2 findings past their 24-hour SLA, by owner.
- Top 10 noisiest policies, with the action taken on each (scoped, re-severitied, or kept).
- Number of accepted risks and expiring exceptions.
- Runtime coverage percentage (hosts and containers with a Defender) so a detection gap shows up before an incident does.
Cadence. Weekly 30-minute review of the top noise sources and any tier-1 misses; monthly report to the CISO; quarterly re-test of the canaries with results attached. Tie every metric to an action someone owns, otherwise the report becomes another channel nobody reads.