RPO Promises Versus Protection Domain Reality

The real RPO today is 4 hours, not 30 minutes.

• Snapshots are local to the source cluster. If the site is lost, local snapshots are lost with it. A 30-minute snapshot schedule protects against logical corruption and accidental deletion on the source - it does nothing for a site failure.
• The only copy that survives a site loss is the replicated one, and replication runs every 4 hours. Worst case is therefore just under 4 hours of lost writes, which matches what the test showed.

Options for 15 minutes, in order of cost:

1. Shorten the async replication interval to 15 minutes. This is the cheapest fix but must be validated: the delta must actually transfer within the window. A 4-hour schedule may already be carrying large deltas; if change rate is high the job will not complete in 15 minutes and you will get a permanently lagging, never-quite-current replica. Measure change rate and available bandwidth first.
2. Near-synchronous / low-latency replication, if round-trip latency between sites is low enough. This gets RPO to near-zero but constrains latency, bandwidth and the DR site's fit-out.
3. Tier the requirement. 140 VMs rarely all need 15 minutes. Put the order database and financial systems on the tightest protection, and keep the rest at an interval the link can sustain. A flat promise to the business that everything is 15 minutes will fail at the first real test.

Write the number down with its definition (async, 15-minute interval, measured lag) rather than agreeing to '15 minutes RPO'.

It cannot be protected as configured. The DR target has ~5.4 TB free; the VM needs 6 TB of replica capacity. It will either fail to replicate or consume the remaining headroom and destabilise the DR cluster - and either way it is not protected today despite appearing in the protection domain.

Reasoning to apply:

• Protection-domain membership is not the same as successful replication. Check the replication status and last-successful time per VM, not just the schedule.
• The capacity question is not 'does the first full replica fit' but 'does the replica plus its growth fit, with headroom, for the retention period'.
• Large VMs also make the 15-minute target harder: a 6 TB disk with any meaningful change rate cannot ship its delta in 15 minutes over a typical WAN.

Handling:

1. Move the VM to its own protection domain with a target that has the capacity, or expand the DR cluster before it is added back.
2. Fix capacity before - not after - the next replication cycle, so you do not discover the failure during an actual failover.
3. Report it to the business explicitly: at present this VM has no off-site copy, so its real RPO is 'undefined', not 4 hours.

Why it broke down:

• AOS needs working headroom on every node for metadata, Curator activity and new extent allocation. Approaching full, the cluster cannot place writes, Curator work stalls, and a CVM can stop responding - which is exactly the sequence described.
• Powering on 40 VMs also added snapshot and write activity on top of the resident replicas, so usage didn't just sit at the replica size; it climbed.
• The alerts and the unresponsive CVM were symptoms of the same condition, not separate problems.

The sizing mistake: the DR cluster was sized to hold replicas. It needs to hold replicas plus the running footprint of whatever you intend to power on, plus headroom for RF2 rebuild after any node loss during the failover.

Sizing method to propose:

1. Define the failover runbook: which VMs power on, in what order, and at what tier. Not 'all 140 at once'.
2. Size for the peak set that will actually run, at RF2, with meaningful free capacity kept in reserve - do not plan to operate at 90%+ after failover.
3. Test with the runbook, not with an ad hoc selection of 40 VMs, and watch usage during the test rather than after.
4. Re-check after every protection-domain or workload change; DR capacity silently drifts out of date otherwise.