DevSecOps Foundation study materials
DevSecOps Foundation study resources

1,400 Alerts a Week and a Nine-Day Dwell Time

DevSecOps Foundation · Put your knowledge to work in context.

All case studies

1,400 Alerts a Week and a Nine-Day Dwell Time

Atlas Retail's security operations center has three analysts covering 24/5. The SIEM ingests 1.8 TB per day, 92% of it verbose application debug logs. Analysts receive about 1,400 alerts per week; 87% are closed as benign, and the top rule fires 600 times weekly on a known scanner. Mean time to acknowledge a phishing report is 11 hours. A recent intrusion via a compromised SaaS token went undetected for nine days; the only signal was a spike in API calls that was logged but not alerted. The cloud team wants to cut logging costs by 40% by reducing retention to seven days for all logs, including authentication and audit logs. Legal requires one-year retention for certain records. The CISO asks how to improve detection without hiring.

1. Assess the detection program and prioritize the fixes. What is the core problem?

Read the worked answer

The core problem is not a lack of logs; it is a lack of detection engineering. The SIEM is storing debug volume, but the highest-value signal in the actual intrusion, anomalous SaaS API call volume, was never turned into an alert. Meanwhile, 1,400 weekly alerts with an 87% benign rate train analysts to dismiss. The top rule firing 600 times on a known scanner is noise that consumes attention and hides real signals. Three analysts cannot triage their way out of a tuning problem.

Prioritize in this order. First, retire or allowlist the scanner rule with an expiry date and route scanner traffic to a separate low-priority queue. Second, build detections for identity and SaaS abuse: token use from a new ASN or country, impossible travel, sudden API call volume against a per-user baseline, and access to unusual object stores. Third, tune alerts by adding asset criticality and identity context so analysts see user, service, and data sensitivity. Fourth, set severity and response targets, and route phishing reports into the same workflow with an 11-hour acknowledgement target reduced to two hours.

Do not start with a new tool. Start with a use-case list tied to the threats that matter: SaaS token theft, cloud key misuse, and data access anomalies. Measure detection coverage and false positive rate per use case, not total alert count.

2. Design a logging and retention strategy that meets legal needs and cuts cost without destroying detection.

Read the worked answer

Separate logs by purpose, not by uniform retention. Security and audit logs, including authentication, authorization, cloud control-plane, and SaaS audit events, should remain searchable hot for 30 days, warm for 90 days, and retained for one year or longer in immutable storage to meet legal requirements. Application debug logs should be reduced at source: lower verbosity in production, sample high-volume debug lines, and retain only 7-14 days hot. That is where the 40% cost reduction should come from, not from authentication and audit logs.

Redact secrets, tokens, and unnecessary personal data at ingestion. Apply field-level access controls so analysts can investigate without broad exposure of customer data. Log access to the log platform itself, and keep legal-hold capability. Define a retention schedule with legal and privacy, and document exceptions. For the SaaS token incident, ensure API gateway and identity provider logs retain enough fields to reconstruct user, source IP, token ID, resource, and action.

Cost controls should be transparent: report ingest by source and log type, charge back verbose debug logging to the producing team, and require a data owner for each new high-volume source. The CISO can then defend a 40% cut that removes debug noise while preserving the evidence needed for detection, response, and audit.

3. How would you measure improvement and address the nine-day dwell time with the current team?

Read the worked answer

Measure dwell time as the primary outcome, supported by mean time to detect, mean time to acknowledge, and mean time to contain. Track false positive rate and alert volume per use case, not as a global number. Add detection coverage for the attack path that succeeded: SaaS token theft, anomalous API volume, and unusual data access. A useful target is to detect the known token pattern in under one hour and to acknowledge a phishing report in under two hours, with weekly review of misses and near misses.

With three analysts, do not add more dashboards. Run a focused purple-team exercise against the SaaS token scenario, instrument the expected signals, and fix the detections that fail. Assign one analyst as detection owner per week, rotating, so tuning is part of the job rather than an afterthought. Move known-scanner noise out of the main queue and automate enrichment for identity and asset context. Create a simple incident playbook for token compromise: revoke sessions, rotate tokens, scope affected data, and notify.

Finally, report the nine-day incident as a detection engineering defect, not an analyst failure. The signal existed in logs; the program failed to alert on it. That framing keeps the team focused on building coverage and avoids a blame cycle that would discourage reporting.