Backup Failures and a Forensic Question After Four Weeks of Enforcement
The PCI-DSS scope of 210 workloads has been in enforcement with default-deny for four weeks. Agents are installed everywhere and flow visibility is agent-based; there is no NetFlow export for this segment. The nightly backup of six database servers now fails for two of them: the backup server 10.20.5.14 is recorded as dropped when it connects to db-03 and db-06, while db-01, db-02, db-04 and db-05 succeed. Nothing else in the scope reports a problem, and the application teams are pushing to set the scope back to monitor mode so backups complete tonight. Separately, incident response has asked a different question about the same window: did host web-01 (10.20.9.31) communicate with any external address on TCP 4444 or 8080 in the last 30 days, and can you prove it? Flow retention is 30 days and the forensic data set has not been exported.
1. Diagnose the backup failure without disabling enforcement, and state what evidence in the flow data tells you the two hosts differ from the other four.
Read the worked answer
Do not revert the scope to monitor. Reverting stops all drops for 210 workloads to fix two, and it also destroys the evidence you are about to need - the same flow records the IR team wants.
Go to flow search, filter by consumer 10.20.5.14 and provider db-03/db-06, and compare against a working pair such as db-01. The comparison is the whole method: the difference between working and failing pairs is the policy gap. Typical findings, in order of likelihood: the failing databases listen on an additional port or a port range (for example a second listener for a different backup plug-in), the backup server connects to a different service address such as a cluster VIP or a secondary NIC on those two hosts, or the database was rebuilt and its identity/tag changed so the provider role no longer matches the rule.
Use the drop record's detail to read the evaluated policy and the reason for the deny - it names the rule and the intended policy, so you can tell 'no rule matched' from 'a deny rule matched first'. That distinction changes the fix entirely.
The remedy is a single, minimal, owner-approved rule addition (consumer 10.20.5.14 to the provider role on the specific port), applied to the workspace, reviewed, and deployed. If urgency is real, an alternative is to exclude just those two providers from the enforced scope temporarily - a much smaller blast radius than a full monitor rollback - but that must be time-boxed and tracked, because exclusions become permanent by default.
2. How would you answer the web-01 forensic question with agent-based flow data, and what are the limits of that answer?
Read the worked answer
Run a flow query filtered on the workload's address (both as consumer and as provider, since external traffic can be either direction), on TCP destination ports 4444 and 8080, over the retention window, and include external/Internet addresses. Agent-based visibility is a strength here: agent flow records include process and user attribution for the workload, so if web-01 initiated the connection you can say which binary and which user did it, which NetFlow on a switch could never tell you.
Then be explicit about what the data does and does not prove. It proves the workload's kernel observed a connection attempt with those attributes, at that time, with that process. It does not prove payload contents - there is no packet capture in the flow record - so you cannot say what was transferred or whether the connection succeeded on the far side. It does not cover traffic in the retention gap, and if a workload's agent was down or the flow was dropped before the agent saw it, absence of a record is not proof of absence. State the retention boundary explicitly: 30 days means the answer covers 30 days and no more.
Before handing anything to IR, export the query result with timestamps and query parameters so the result is reproducible, record who ran the query and when, and note the cluster's time source. Do not hand over a screenshot. If IR needs payload evidence, the answer is host-level capture or endpoint telemetry, initiated through the proper forensic process, not re-querying the same flow data.
3. What changes to policy lifecycle and operations would prevent both of these problems recurring next quarter?
Read the worked answer
The backup failure is a change-management failure as much as a policy failure. Any database rebuild, listener change or VIP addition in an enforced scope must trigger a policy review; that means the ticketing workflow must be wired to the workspace, either through an approval step before deployment or a post-change review of the drop list. In parallel, keep a scheduled weekly review of the would-be-drops and actual drops report for each enforced scope, owned by a named person with a documented response time. Two hosts failing silently for a night is tolerable; two hosts failing for a quarter is not.
Before each enforcement increment or quarterly cycle, run policy simulation against the full retention window rather than the ADM window only - this is what catches the monthly and quarterly workloads.
For forensics, decide retention deliberately. Thirty days is a choice with a cost; if investigations routinely ask about 90 days, raise flow retention for the PCI scope and size the disk for it, or add an export to a long-term store. Also pre-agree the export and evidence-handling procedure so an IR request is a routine task with a known turnaround, and make sure at least two people can run the query. Finally, monitor agent coverage continuously - the strongest forensic answer in the world is worthless if 60 hosts were unmonitored for part of the window.