Northwind Logistics: a first-ever regional failover with a 1-hour SQL RTO
Northwind Logistics runs a three-tier .NET application on 60 IaaS VMs and 4 SQL Server 2019 VMs holding 2 TB, all in Sweden Central. The business requires an RTO of 4 hours for the web tier and 1 hour for SQL, with an RPO of 15 minutes for SQL and 24 hours for the 20 TB file share. A full regional outage must be survivable. Compliance requires 35 daily, 12 monthly, and 7 yearly backup copies held in a different region from production, and those copies must not be deletable by a compromised administrator for at least 30 days. The budget will not support active-active, and the team has never performed a failover.
1. Choose the replication technology for each tier, with justification against the stated RTO and RPO. Explain why one technology cannot cover everything.
Read the worked answer
Per-tier technology, because the RTO/RPO pairs differ by an order of magnitude.
- Web/app tier (60 VMs, RTO 4h): Azure Site Recovery, VMs replicating to the paired region (Sweden South) with crash-consistent recovery points every 5 minutes. Group them in a recovery plan with an ordered start sequence and automation runbooks for DNS and configuration. Four hours suits cold infrastructure plus ASR; you do not need warm standby.
- SQL (RTO 1h, RPO 15 min): ASR is the wrong tool. Crash-consistent VM replication gives no transaction-level guarantee, and SQL may not recover cleanly. Use an Always On availability group with an asynchronous secondary in the secondary region. Asynchronous failover is manual (forced), which fits a 1-hour RTO, and the RPO is seconds to a few minutes - comfortably inside 15 minutes. Keep the AG listener in DNS so connection strings do not change at cutover.
- File share (RPO 24h): RA-GRS storage with an account failover; the last sync point is typically within about 15 minutes, which beats the 24-hour RPO without extra machinery.
- Front door: Traffic Manager or Front Door in front of the web tier, with health probes, so the cutover is a profile change rather than a DNS scramble.
2. Design the backup and vault configuration so the regional and 30-day-tamper requirements are both met, and explain the settings that are off by default.
Read the worked answer
A vault that survives the region and resists its own administrators.
- Vault placement. A Recovery Services vault with GRS replication stores backup data in the paired region. Place the vault resource in Sweden South, or keep it in Sweden Central and enable cross-region restore - which is off by default and only settable after the first backup has run. Without it, 'the data is in the paired region' is not the same as 'you can restore from it'.
- Policy. One policy with daily retention of 35 days plus weekly, monthly (12), and yearly (7) rules. Azure Backup retention rules compose; do not build four separate policies.
- Anti-tamper. Enable vault soft delete (extend beyond the 14-day default) and vault immutability (locked), then configure multi-user authorization with a Resource Guard in a different subscription or resource group so that the workload administrators cannot disable soft delete or purge recovery points. That is the control that answers 'a compromised administrator', because that administrator will hold Contributor on the vault's resource group.
- Scope the workload subscription. Separation of duties only works if backup operators do not also administer the protected resources.
3. The team has never failed over. Define what 'proven' means here, and what you would instrument to defend the 4-hour and 1-hour RTO claims.
Read the worked answer
An untested recovery plan is a hypothesis, not a design.
- Site Recovery test failover into an isolated VNet with no production impact: proves the recovery plan ordering, the VNet mappings, the automation account, and the DNS scripts. Run it after any material change and clean up with 'cleanup test failover' so replication stays healthy.
- SQL failover drill: forced manual failover to the async secondary, measure actual time and data loss, then fail back. Confirm the listener and the application connection string work unchanged.
- Backup restore drill: restore a SQL database to an alternate name and a file share to an alternate location; record elapsed time from request to usable data - that number, not the console's green tick, is the real RTO.
- Instrument: record start/end timestamps for each drill, store them next to the plan, and alert on replication health (
ASR replication health, AG synchronization state, backup job failures). RPO is measured by the gap between the last recoverable point and the incident, so track that during drills, not after a real event. - Human dependencies: name an incident commander, document who can run the failover, and store credentials in a vault reachable during a regional outage.