The Alert Storm at 03:40

First, establish whether this is an attack or a broken job, because the response differs completely. The fastest discriminator is the 50 successes: failed logons alone are often a misconfigured service, but successful authentication to 12 servers from an unexpected host is the part that matters.

Order of work:

1. Confirm the source host identity - is 10.20.4.31 a known backup server, a new build, or unknown? Check asset inventory and DHCP.
2. Check what svc_backup normally does at this hour and whether a scheduled job changed. A backup job that was repointed to a new host explains both the failed and successful logons.
3. Look at the successes specifically: which 12 servers, what happened after logon (process creation, service installs, file access). Success followed by lateral movement is the escalation trigger.
4. Check whether the account's password was recently changed or the job's credentials rotated - a stale credential causes exactly this failure pattern.

The judgement: do not start by tuning the rule down. 20 minutes is enough to answer "new host or compromised host" and "job change or intrusion." If the successes show no post-logon activity beyond the backup agent, treat as operational and hand over with notes; if they show anything else, escalate immediately.

The situation has changed from operational noise to probable credential compromise of a privileged service account, used from an unmanaged device.

Containment sequence:

1. Disable or reset svc_backup credentials - but coordinate first, because backups may fail and that is a business risk. Get the backup owner on the call.
2. Isolate 10.20.4.31 from the network at the switch port or via NAC, not by asking the user to shut it down, so state is preserved.
3. Identify the user and how they obtained the credentials - shared documentation, a script with a hardcoded password, a previous incident.
4. Review the 12 servers for anything the account did beyond normal backup behaviour, and check the other 28 for partial or failed access.

Preserve: SIEM raw events for the window (export before retention rolls), the laptop's memory and disk image, authentication logs from all 40 servers, and the backup job configuration history. Note the 20-minute constraint: you will not finish this before handover, so the priority is stopping the credential from working and capturing the laptop before it is touched. Write the handover so the next analyst knows what is contained and what is still open.

Judge the rule on whether it produced actionable signal at a cost the team can bear.

What it did well: it fired on a genuine anomaly - a service account authenticating from an unexpected host at an unusual volume. That is exactly the behaviour a threshold rule should catch, and it did so without needing a fancy detection.

What it did badly: 1,150 near-identical alerts for one root cause created noise that could have buried the 50 successes. If the analyst had triaged by volume rather than by meaning, the important alerts would have been last.

The fix is not to raise the threshold until it stops firing - that would have missed this. Better options: correlate failed and successful logons for the same account and source into a single higher-fidelity alert; enrich with asset ownership so an unknown host is flagged; and suppress duplicates within the incident rather than at the rule. Then test the revised rule against this exact scenario to confirm it still fires, ideally once, with the successes attached. Document the change and the reasoning so the next analyst understands why the rule looks the way it does.