Promising a 15-Minute RPO You Cannot Measure
An internal platform runs 40 application nodes in one region with a 2.4 TB primary database and 6 TB of object storage. The business has agreed to a 15-minute RPO and a 2-hour RTO. Regulators require that a recoverable copy remain inside the country, and the only approved secondary region is 900 km away with roughly 35% spare compute. Today the database streams asynchronously to a standby that keeps pace off-peak; during month-end, measured replica lag peaks at 22 minutes. The standby has never been promoted in a test, because the runbook contains 40 manual steps. DNS records for the platform carry a 300-second TTL. The ops team proposes keeping the current design and 'just documenting the failover better.'
1. Is 'keep the design, document failover better' defensible against a 15-minute RPO and a 2-hour RTO? Show your reasoning.
Read the worked answer
No - it fails both targets, for different reasons.
RPO. The measured lag during month-end is 22 minutes against a 15-minute commitment. Documentation does not change replication lag; only the replication mode or the write path does. Note when the gap appears: off-peak the standby is current, and the miss happens precisely at month-end, when losing 22 minutes of financial postings is least acceptable. A failover at the wrong hour silently violates the contract.
RTO. Two hours is 120 minutes. Take the 40 manual steps at a generous 1.5 minutes each with no surprises: 60 minutes. Add detection and the decision to invoke, database promotion, application validation, and DNS propagation at a 300 s TTL - plus clients and resolvers that cache past the TTL. That leaves almost no contingency, for a procedure with 40 opportunities to hit an undocumented dependency. And because the standby has never been promoted, the 2 hours is an estimate, not a demonstrated capability.
The proposal is a documentation change applied to a design gap. Either replication changes so the RPO holds under load, or the RPO is formally renegotiated with the business and the exception is signed.
2. What would you change, ranked by impact per unit of effort? What does 35% spare compute actually buy you?
Read the worked answer
Ranked:
- Instrument lag as a first-class SLO. Emit replica lag continuously with alerts at 5 and 10 minutes. Today the 22-minute breach was discovered incidentally; you cannot claim a 15-minute RPO you do not measure, and you cannot tune what you do not see.
- Turn the runbook into an executable pipeline. Promote, re-point, verify, roll back - idempotent, versioned, tested. Cheapest RTO win available, because 40 manual steps is a variance problem, not a capacity problem.
- Close the RPO gap structurally. Either quasi-synchronous commit for the write paths that matter, or tier the RPO explicitly (15 min for transactions, 1 hour for reporting) and get it signed. Do not leave a single number covering workloads with different tolerances.
- Prove it on a schedule. A game day that actually promotes a copy and times it, plus a periodic restore test of object storage - which is often the untested half.
On 35% headroom: 40 × 1.35 ≈ 54 nodes' worth of capacity. That only helps if the secondary can run the same node count at reduced per-node throughput via pre-baked images and autoscaling, or if the service degrades acceptably at 40/54 ≈ 74% of normal capacity. If it cannot, 35% is not failover headroom - it is a licence to run a permanently degraded service. Also verify the spare capacity is not in the same failure domain as the standby database; a quota number that evaporates during a regional event is not headroom.
3. What is the smallest test that proves the 2-hour RTO without a full failover?
Read the worked answer
A timed promote-and-verify drill in a segmented environment.
- Snapshot the standby, promote it, and run a fixed verification suite: row counts against source, checksums on ten known tables, and one synthetic transaction through the application tier.
- Measure wall-clock from decision to verified, not from command to command. The long poles are usually database promotion, secret and config availability in the secondary, and service discovery - not the copy itself.
- Measure DNS separately with a throwaway record so the 300 s TTL and client caching behavior are observed rather than assumed. Many teams discover a hardcoded address or an ignore-TTL client here.
Decision rule: if the drill plus a 30% contingency lands under 120 minutes, the RTO is credible. If it does not, the drill has told you precisely which step to automate next - which is more valuable than a full failover, because a full failover tests everything at once and attributes nothing.