A Nine-Hour Outage: Closing the Gap Between Stated RTO/RPO and the Design
Tailspin's order system runs in West Europe: an App Service plan with six instances, Azure SQL Database Business Critical zone-redundant, Service Bus Premium, and a Storage account using RA-GRS. A regional incident last quarter took orders down for nine hours. The business now requires a 4-hour RTO and a 15-minute RPO for order data, plus no silent loss of submitted orders. The team has geo-restore configured for SQL, routes every platform metric into a single action group that emails one on-call engineer, and logged 3,100 alerts last month against two real incidents. There is no written failover runbook, and the disaster recovery subscription mirrors production naming but has never been exercised.
1. Which components cannot meet the 4-hour RTO and 15-minute RPO, and what should replace them?
Read the worked answer
Take each tier against the numbers.
- SQL: geo-restore restores from a geo-redundant backup into a new database. Its recovery time is measured in hours and its data loss in tens of minutes, so it fails both targets. Replace with an auto-failover group to a secondary region: a read-write listener endpoint, RPO typically seconds to low minutes, and a failover that completes in minutes. Business Critical already gives zone redundancy inside the region, which protects against datacenter failure but not region failure.
- App Service: there is no automatic cross-region failover. Run a passive (or scaled-down active) stamp in the paired region and front it with Azure Front Door or Traffic Manager using health probes, so clients hit a stable hostname and failover is a routing decision, not a DNS TTL gamble.
- Service Bus Premium: geo-disaster recovery pairs namespaces but replicates metadata, not messages. In-flight messages in the primary are lost on failover. That directly violates the no-silent-loss requirement, so the design needs an outbox or idempotent replay on the producer side rather than trusting the platform.
- Storage RA-GRS: replication is asynchronous, so the secondary is behind and account failover is a manual, potentially lossy operation. Treat it as a backup of last resort, not as the RPO mechanism.
The honest summary: two of four tiers were designed for the wrong failure mode, and one cannot satisfy the requirement without application changes.
2. Redesign the alerting so that 3,100 alerts a month becomes a signal the on-call engineer trusts.
Read the worked answer
The current design is a firehose because one action group receives every metric. Start from the failure the business cares about and work backwards.
- Alert on symptoms, not causes. Users cannot feel CPU percentage. Alert on HTTP 5xx rate, p95 latency against the SLA, queue depth and age of the oldest message, and failed order submissions. Keep a small set of resource-health and saturation alerts for capacity planning, routed to a ticket queue rather than to a phone.
- Severity tiers with different action groups. Sev1 pages the on-call engineer; Sev2 creates a ticket; Sev3 goes to a dashboard only. This alone removes most of the volume.
- Use the right signal type. Metric alerts are cheap and near-real-time for thresholds. Log alerts are for queries that need joins or text, and should run at the longest frequency that still satisfies detection time.
- Suppress and deduplicate. Action groups support common alert schemas and alert processing rules for maintenance windows; use them so a single node reboot does not produce forty notifications.
- Add synthetic availability tests from multiple regions, which detect a bad deployment before the first customer complaint does.
- Review monthly: for every page, ask what action it prompted. If none, delete it or downgrade it.
3. How would you prove the 4-hour RTO before the next real incident, and what would you fix first in the runbook?
Read the worked answer
Prove it in a scheduled game day in the paired region, measuring wall-clock time from the declared start to verified order throughput, not from the moment the database finished failing over.
Runbook items to write and test first, in this order:
- Decision authority and trigger criteria. Who declares a disaster, and what evidence is required. Nine hours was likely a coordination failure as much as a technical one.
- Exact failover sequence and expected durations per component, with the SQL failover group's forced versus planned failover semantics spelled out; forced failover accepts potential data loss, planned failover does not, and the operator must know which one the scenario calls for.
- Front Door origin switch and validation steps, including how to confirm the app is actually serving from the secondary and not just returning a cached error page.
- Service Bus alias cutover and the replay procedure for messages that were in flight.
- Data integrity verification after failover: order counts, sequence continuity, and reconciliation of anything queued at the moment of failure.
- Failback strategy, which is usually the step teams forget and which often takes longer than the failover itself.
Automate the repetitive parts with runbooks or pipelines, but keep a human approval gate on the destructive ones. Then repeat the drill quarterly, and after every real incident, and record the actual RTO each time so trend replaces optimism.