The Immutable Window That Started After the Intrusion
Northline Freight operates 1,400 servers across three dispatch hubs. Its backup platform keeps 30 days of nightly synthetic fulls and 14 days of immutable restore points on a hardened repository. Ransomware is detected on 300 servers; EDR telemetry shows initial access 26 days earlier. The backup server was domain-joined, and the backup service account had local administrator rights on the repository. The attacker deleted the backup catalog and any immutable snapshot older than 14 days. Backups from the last nine days are encrypted because immutability was applied only to weekly fulls, not to incrementals. Weekly tapes go to an offsite vault; the most recent tape is 19 days old. A monthly tape from 33 days ago is also vaulted, but its index lived on an encrypted server. The tape library still has barcode inventory, and the vendor says it can rebuild the catalog. Dispatch operations run on an ERP with a 12-hour RTO and a 2-hour RPO.
1. Determine which recovery points are trustworthy and sequence the recovery. What do you tell the business about the RPO gap?
Read the worked answer
The only point that predates the known initial access is the 33-day-old monthly tape, assuming the 26-day EDR timeline is reliable. The 19-day tape and the 14-day immutable restore points were created after the attacker was present, so they are not trusted for integrity even if they are readable. The last nine days of backups are encrypted. That means the earliest defensible recovery point is 33 days old, not the two-hour RPO.
Sequence the work in this order. First, isolate the backup infrastructure and preserve it for forensics; do not let restoration overwrite evidence. Second, rebuild the tape catalog from the barcode inventory and vendor support, but verify the tape's chain of custody and hash the restored images. Third, restore to a clean, isolated environment with new credentials, not to the compromised domain. Fourth, validate the ERP and dispatch data against external evidence such as bank feeds, EDI transactions, customer confirmations, and paper dispatch records. Only then reconnect to production.
Tell the business plainly that the clean recovery point is 33 days old and that the two-hour RPO cannot be met. Offer a choice: reconstruct the last 33 days from external records at high manual cost, or accept a shorter but unverified 19-day point only after forensic scanning and business reconciliation. Do not call the 19-day tape clean without evidence.
2. What control changes would you prioritize so this failure cannot repeat, and how do you justify the order?
Read the worked answer
Prioritize by the failure chain observed, not by a generic backup checklist. The attacker reached the backup server because it was domain-joined, and the service account had local administrator rights on the repository. Fix identity separation first: remove the backup infrastructure from the production domain, use dedicated backup credentials, enforce MFA for administrative access, and apply least privilege so the backup service cannot delete immutable data or the catalog. Second, make immutability apply to every restore point, including incrementals, and extend retention beyond the observed attacker dwell time, for example 60 to 90 days. Third, keep an offline or air-gapped copy with a catalog that is not stored only on a network server. Fourth, monitor and alert on snapshot deletion, catalog changes, and backup job failures. Fifth, test restores quarterly from the immutable and offline copies into an isolated environment.
Justify the order by time to reduce the same failure: identity separation and immutability for all restore points close the exact paths the attacker used. Longer retention addresses the 26-day dwell time. Offline catalog copies address the recovery dependency that nearly blocked the tape restore. Restore testing proves the controls work. Cost the items and present them as risk reduction against the 33-day data loss exposure, not as backup modernization.
3. How would you prove to auditors and the business that the restored ERP data is complete and unaltered?
Read the worked answer
Define a validation plan before restoring. Record the source tape, hash, restore time, operator, and target environment. After restore, scan the recovered images and the restored systems for the known malware indicators and for unauthorized accounts, scheduled tasks, and remote-access tools. Then run three levels of reconciliation: record counts by table and date against external reports; financial control totals such as invoices, payments, and general ledger balances against bank and EDI records; and business sampling of dispatch orders and customer records against source documents.
Obtain sign-off from the ERP process owner and finance, not only from IT. Document any unexplained variance, the investigation, and the resolution. If the restored data cannot be reconciled, do not put it into production. Keep the evidence package with the incident record and legal hold. This gives auditors a defensible trail from tape to production and gives the business a stated level of confidence rather than an assertion that the restore succeeded.