The 30-Minute RTO That the Design Cannot Deliver
Northwind Analytics runs a SaaS reporting product from a single region. The web tier is stateless across three availability zones behind a load balancer, but the application tier keeps user sessions in local memory and depends on load-balancer sticky sessions. One relational database primary handles all writes, with a single read replica in the same region. Backups are a full nightly snapshot at 02:00 with no transaction log shipping. Sales has promised customers an RTO of 30 minutes and an RPO of 5 minutes, and the board wants a documented cross-region DR story before the next contract renewal. Monthly infrastructure spend is capped at a 25 percent increase over today. Last quarter a 40-minute availability-zone impairment caused 22 minutes of failed logins because sessions were lost.
1. Identify which parts of the current design make the 30-minute RTO and 5-minute RPO unachievable, and explain the mechanism behind each failure.
Read the worked answer
Start from the definitions, not the component list. RPO and RTO are properties of a recovery mechanism, not of a backup schedule. Ask for each tier: what artifact do I restore, how long does it take, and how much data is lost at the moment of failure?
| Component | Why it breaks the target |
|---|---|
| In-memory sessions + sticky sessions | Failover to another instance or AZ logs every pinned user out. The 40-minute AZ event already demonstrated 22 minutes of failed logins. This is an availability defect, not a DR one, and it will show up at a much smaller blast radius than a region loss. |
| Nightly full backup only | Worst-case data loss is 24 hours, not 5 minutes. RPO of 5 minutes requires continuous log/WAL shipping or point-in-time recovery, with a verified restore. |
| Single primary, single region | A region loss means the primary is gone and there is no running replacement. Rebuilding the app tier plus restoring a multi-terabyte database from object storage is a multi-hour operation, not 30 minutes. |
| No documented, rehearsed runbook | Untested recovery is an estimate, not an RTO. The first real invocation typically takes several times the paper figure. |
Reasoning to state explicitly: the RPO gap and the RTO gap have different fixes. Log shipping closes the RPO gap cheaply. The RTO gap needs pre-existing capacity - either a warm standby or infrastructure-as-code that can rebuild fast - because provisioning and bootstrapping dominate the 30-minute window.
Order of work: session externalisation first (cheap, removes an entire failure class), then continuous log shipping (largest RPO gain per dollar), then the regional standby.
2. Recommend a target architecture that meets both objectives within the 25 percent budget increase, and state what you would cut if the budget did not stretch.
Read the worked answer
Meet RPO first, because it is the cheap half.
- Enable continuous WAL/log shipping to object storage in the second region with point-in-time recovery. This moves worst-case loss from 24 hours to under 5 minutes and costs storage plus a small amount of egress.
- Keep an asynchronous cross-region read replica as a warm standby. Note the honest trade-off: asynchronous replication means a real failover can lose the last few seconds of committed transactions - that is compatible with a 5-minute RPO, not with zero.
Meet RTO with warm capacity, not with hope.
- Stand up the standby region as a pilot light: database replica running, application tier defined entirely in infrastructure as code but scaled to zero. Scale-up is minutes of automation, not hours of rebuilding.
- Externalise sessions into a replicated managed cache or move to signed stateless tokens, so no tier depends on which instance served the previous request. This also removes sticky sessions.
- DNS with a short TTL plus health checks, and an automated or one-click promotion runbook that includes fencing the old primary so you cannot get split-brain writes.
- Rehearse quarterly and record the measured time. An untested 20-minute estimate is not an RTO.
Budget arithmetic to show: a always-on duplicate region roughly doubles compute and is what blows the 25 percent cap. A pilot light is typically 10–20 percent of production spend - replica instance, storage, replication egress, and the object storage for logs. That fits.
If the budget still does not stretch, cut in this order:
- Drop the always-running application tier in the standby region (keep IaC).
- Shorten backup retention in the primary region rather than weakening replication.
- Do not cut the log shipping or the rehearsal - those are the two cheapest items with the largest effect on the stated objectives.
State the residual risk you are accepting in writing, and get the business to sign it.
3. If you could only do one thing this quarter, what delivers the best risk reduction, and what new risk does it introduce?
Read the worked answer
The single highest-value change is externalising session state - moving it out of instance memory into a replicated cache or eliminating it with signed tokens.
Why it beats the regional work this quarter:
- Frequency. Region loss is rare; instance replacement, deployments, and AZ impairments are routine. The last AZ event cost 22 minutes of failed logins, and the same defect will fire on every rolling deployment.
- It is a prerequisite for the DR target anyway - a warm standby that loses all sessions on cutover is not a working recovery.
- Cost is low, and it removes sticky sessions, which otherwise make autoscaling and blue/green painful.
The new risks it introduces, stated plainly:
- The cache becomes a shared dependency. If it is single-node, you have replaced ten small failure domains with one large one. It must be replicated across AZs with a fallback path.
- Session data now crosses the network, so it needs encryption in transit and a sane TTL.
- If you choose signed tokens instead, you inherit revocation problems: a stolen token stays valid until expiry unless you build a revocation list. For an analytics product with long sessions, that is a real trade-off - a cache with short TTLs is often the better answer.
How you would know it worked: rolling deployments cause zero authentication failures, an instance termination during peak produces no login errors, and the cache's own availability is monitored as a tier-one dependency.