A Clean Restore That Cannot Yet Be Called Clean
Vantage Retail's e-commerce platform was restored from backup in 14 hours after a web shell was found on a front-end server. The site is processing orders, but forensics shows the attacker had access for 11 days and could alter prices, customer records, and order status. The database was restored from a snapshot taken 36 hours before detection, yet the attacker may have modified data earlier. Point-of-sale systems were not restored and continue operating. PCI DSS evidence, a legal hold, and a possible class action mean the firm must prove the restored environment is clean. Web and database logs retain only seven days. The attacker's tooling is still being analyzed. The CISO wants to close the incident in 48 hours and skip re-restoration; the CIO wants to restore from a 12-day-old backup to be safe. The business says a second outage would cost $180,000 per hour.
1. Design a validation plan for the restored e-commerce environment and its data.
Read the worked answer
Start with a trust boundary. The restored platform is a suspect environment until proven otherwise. Preserve forensic images of the compromised front end and the database before further changes. Rebuild the front end from known-good infrastructure as code, rotate all credentials and keys, and scan for the web shell and related tooling. Confirm no attacker-controlled accounts, scheduled tasks, or outbound connections remain.
Validate data at three levels. Technical: compare row counts, checksums, and referential integrity against the backup and against external payment records. Business: reconcile orders, payments, refunds, inventory, and customer records to the payment processor and fulfillment systems; sample price changes and order status transitions; look for unauthorized discounts or altered shipping addresses. Security: review administrator activity, new accounts, and database triggers or jobs. Because logs retain only seven days, use external evidence and immutable payment records to cover the 11-day exposure.
Define pass criteria and an owner before starting. If anomalies appear, quarantine the affected data and investigate rather than patching in place. Produce a signed validation report for PCI, legal, and the business. Closing the incident in 48 hours is acceptable only if the validation evidence supports it; the calendar is not the criterion.
2. Should the firm re-restore from the 12-day-old backup? Give a decision rule.
Read the worked answer
Not automatically. The attacker had access for 11 days, so a 12-day-old backup is close to the boundary but not proven clean; it may predate access, or the timeline may be off by hours. Re-restoring also imposes at least $180,000 per hour of outage and may lose valid orders. The decision should depend on evidence of tampering in the current restore, not on the age of the backup alone.
Use a decision rule: if validation finds unauthorized data changes, malicious database objects, or an inability to reconcile financial records, rebuild in a parallel environment from the oldest backup that predates the earliest confirmed attacker activity, validate it, and cut over with a planned window. If validation passes, keep the current restore and monitor. The CIO's caution is reasonable, so test the 12-day backup in parallel without taking the site down. Compare the cost of the parallel test with the cost of an unplanned second outage and with the legal exposure of serving altered data. Document the decision and the evidence.
3. What should the post-incident review cover, and how should it avoid becoming a blame exercise?
Read the worked answer
The review should reconstruct the timeline from initial access to detection, containment, restore, and validation; measure detection time, containment time, recovery time, and data loss; identify the control failures that allowed the web shell and the 11-day dwell time; and assess whether the seven-day log retention limited investigation. Include the restore validation results, the decision not to re-restore, and the cost of the outage. Cover legal and regulatory actions and whether notifications were timely.
Keep it blameless by focusing on conditions and decisions rather than individuals. Ask why the system allowed the action, what signal was missed, and what would make the right action easier next time. Assign actions with owners and dates, and track them like other risk treatments. Include the business and legal teams, not only IT. Publish a short version to the board that states what changed, and retain the detailed evidence under legal hold. The test of the review is whether the next validation is faster and more defensible, not whether someone is held responsible.