The Backup Exclusion That Became a Blind Spot
Contoso Manufacturing runs an EDR agent on 850 Windows servers and workstations. Fourteen months ago the backup administrator, with SOC sign-off, added a path-based exclusion for C:\ProgramData\Veeam\ to stop the agent scanning backup staging files and degrading backup throughput. The exclusion removes that directory from real-time scanning and from file-event telemetry; process-creation telemetry is unaffected. During a routine hunt, an analyst notices that on three servers rundll32.exe has been launching with a command line pointing at a DLL inside the excluded path, roughly every 30 minutes since the previous Tuesday, with the parent process being svchost.exe hosting the task scheduler service. There is no file hash in the console, no signature status, and no network events for those processes, because the endpoint firewall only aggregates per-process byte counts once an hour. The three servers sit in the same VLAN and share a local administrator password last rotated in 2022. Change management has no record of new software on those hosts.
1. Working with only process-creation telemetry, what steps do you take to determine whether this is malicious, and in what order?
Read the worked answer
Treat the missing file telemetry as a collection problem before it becomes a conclusion problem. Order:
- Acquire the artefact, don't infer it. Use live response to pull the DLL, its directory listing, timestamps, and the scheduled task XML from one host. Hash it, check Authenticode signer and whether the certificate chains to a legitimate publisher, and submit to a sandbox and multi-engine reputation. A Veeam-signed component with a valid chain looks very different from an unsigned DLL dropped last Tuesday.
- Establish whether it is fleet-wide or unique. Query all 850 hosts for rundll32 with the same parent path. A backup component appears on backup servers; a backdoor appears on the three hosts the actor reached. Uniqueness is a strong discriminator.
- Rebuild the execution chain from what is left. Task Scheduler creation (4698/4702), service creation (7045), Amcache and Prefetch, WMI event subscriptions, and recent file writes elsewhere on disk - the dropper probably wrote outside the excluded path too.
- Get the network picture the firewall hid. Live response
netstat/connection table and a packet capture on one host for 30 minutes gives you destination, JA3, and beacon interval, which the hourly byte aggregation cannot. - Check credential exposure. Shared local admin password plus three compromised hosts means lateral movement is plausible; query 4624 type 3 and 4648 on other hosts for that account.
Only after steps 1 and 4 fail to explain the activity would I call it benign. Absence of hash data is not absence of malice - it is absence of visibility.
2. How do you remediate the exclusion without breaking backup performance, and what compensating detections do you add?
Read the worked answer
The exclusion was a reasonable performance decision with an unreasonable implementation. Path-based exclusions exclude everything that ever lands in that folder, so anyone who can write there inherits the blind spot - and a staging directory is exactly where a backup service account writes on a schedule.
Replacement design:
- Exclude by signer or hash of the backup binaries, not by directory. Veeam's executables carry a stable publisher certificate; scanning can skip those processes while still scanning anything else that appears in the folder.
- Constrain write access. ACL the staging directory so only the backup service account can create files. A user-context process writing there becomes both impossible and detectable.
- Add file integrity monitoring on the directory, alerting on any file creation whose signer is not the backup vendor, plus any DLL creation at all if the backup product does not legitimately drop DLLs.
- Detect the execution pattern. Alert on rundll32, regsvr32, or svchost-spawned DLL loads from user-writable directories, and on scheduled tasks whose action points into staging paths. These fire regardless of scan exclusions.
- Govern the exclusion list. Every exclusion carries an owner, a business justification, a review date, and a compensating control reference. Review quarterly; 14-month-old exclusions with no owner are the norm and should not be.
- Instrument the performance claim. Measure agent CPU and backup window duration before and after removing the path exclusion so the change can be defended to the backup team with data.
The general principle: never exclude a location an attacker can write to without adding detection that covers the same location by another means.
3. The three servers share a local administrator password last rotated in 2022. How does that change your containment sequence?
Read the worked answer
It changes both timing and scope. Normally you contain in order of severity, one host at a time, verifying as you go. With a shared local admin credential on a Tuesday-to-now dwell time, you must assume credential theft already happened and act concurrently.
Revised sequence:
- Contain all three simultaneously - network quarantine via switch ACL or agent isolation, preserving EDR management traffic so you keep telemetry. Sequential isolation gives the actor time to move off a host you have not reached yet.
- Rotate the shared credential immediately, estate-wide, and treat every host where that account has logged on interactively or over SMB as in scope until disproven. Use LAPS or a managed local-admin solution rather than another manually shared password.
- Hunt the credential use before the attacker retools: 4624 logon type 3 with that account, 4648 explicit credentials, 5140/5145 share access, and SMB scans from the three hosts in the preceding days.
- Preserve before cleaning. Memory capture and disk images on at least one host before reimaging, otherwise attribution and scope estimation rest entirely on process telemetry you already know is thin.
- Check the backup infrastructure itself. The staging directory is written by the backup service account, which is often domain-privileged; if that account is compromised the blast radius is the whole estate, not three servers.
Only after the credential is rotated and lateral movement is ruled out do you return to normal per-host remediation and reimage.