RPO/RTO Sign-off for a 3,000 TPS Order Database
A payment-clearing subsidiary of a logistics group is designing the database tier for an order platform on Huawei Cloud. Peak load is 3,000 TPS at a 70/30 read/write mix; current data volume is 600 GB growing by 20 GB per month. The business has signed off two targets: for loss of a single AZ inside the primary region, RPO = 0 and RTO ≤ 15 minutes; for loss of the entire primary region, RPO ≤ 5 minutes and RTO ≤ 2 hours. The application is MySQL 5.7 compatible, uses several stored procedures, and the DBA team has no distributed-database experience. Delivery window is four weeks. Two designs are on the table. Design A: an RDS for MySQL primary/standby HA instance in the primary region, plus DRS real-time synchronization into an RDS for MySQL instance in the DR region. Design B: TaurusDB with read replicas, replacing the MySQL tier entirely.
1. Assess Design A against each of the two targets. State explicitly which target it satisfies, which it only partially satisfies, and what evidence you would require before signing off.
Read the worked answer
AZ failure (RPO 0, RTO ≤ 15 min). Satisfied in principle: an RDS HA instance places the standby in a different AZ with semi-synchronous replication, so a committed transaction exists on the standby before commit returns to the client - RPO 0 for committed transactions. Failover is automatic, but 15 minutes is a target, not a guarantee, so sign-off requires a fault-injection drill measuring actual detection plus promotion plus application reconnection time, with connection strings and DNS TTL already tuned.
Region failure (RPO ≤ 5 min, RTO ≤ 2 h). Only partially satisfied by Design A as written. DRS real-time synchronization is asynchronous, so RPO equals the replication delay - seconds in steady state, but it spikes during bulk loads, index rebuilds and large transactions. The RPO target is therefore met conditionally: you must alarm on replication delay and cap batch transaction size. RTO is not a property of DRS at all; it is your runbook - detect, stop the source, confirm the DR instance has applied the last relay log, promote it, repoint DNS, and verify. That must be scripted and drilled, otherwise 2 hours is guesswork.
Evidence to require before sign-off: semi-sync enabled and verified; binlog ROW format and GTID confirmed; 72 h of replication-delay metrics showing p99 under 60 s; a timed DR switchover drill; and a documented, tested rollback path.
2. Which failure modes remain after Design A is implemented, and how do you close them?
Read the worked answer
Design A covers instance and region loss, but leaves operational gaps that are exactly what the exam-style post-mortem would find:
- DNS/connection stickiness. TTL too long, or connection pools holding the old endpoint, silently blow the RTO. Close: lower TTL a week ahead of the drill, put the switchover behind a script that flushes the pool, and test from the application, not from a jump host.
- DR instance is untested and unmonitored. A replica that has been quietly lagging or has hit a disk-full condition is not a DR site. Close: daily replication-lag and disk-growth alarms, plus a monthly read-only validation job against the DR instance.
- Backups do not exist in the DR region. If the primary region is lost, your backups are lost with it. Close: CBR cross-region backup copies, and confirm restore time as part of the RTO budget.
- Dependencies outside the database. VPC, subnets, security groups, IAM credentials and KMS keys must exist in the DR region; otherwise promotion succeeds and the application still cannot start.
- Human factors. One engineer knows the runbook. Close: written runbook, two trained operators, quarterly joint drill with the business.
Each item is cheap individually; collectively they are the difference between a paper RPO and a real one.
3. The CTO asks why not simply adopt Design B and skip the complexity. Give your recommendation with reasoning.
Read the worked answer
Design B is architecturally attractive but is the wrong choice for this delivery.
What B genuinely offers: compute-storage separation, fast failover, read replicas that do not consume primary resources, and a much higher write ceiling than a single HA instance - valuable at 3,000 TPS and beyond.
Why it loses here:
- Compatibility risk. MySQL 5.7-compatible stored procedures must be validated on TaurusDB; any behaviour difference becomes an application change inside a four-week window.
- Skill risk. A DBA team with no distributed-database experience will not be competent at diagnosing storage-layer latency or replica anomalies in week one.
- It does not solve the stated requirement. Design B replaces the primary-region database but still needs a cross-region DR mechanism and a rehearsed runbook. The region-loss RTO problem is unchanged.
- Cutover risk compounding. Migrating engine and migrating to cloud in the same window doubles the failure surface with no rollback path.
Recommendation: ship Design A now, with the DRS lag alarms and a drilled switchover, because it meets the signed-off targets at known risk. Run B as an explicit v2 evaluation with a compatibility test suite and a training plan. The judgement being tested is that a technically superior option can still be the wrong one when the binding constraints are time, skills and rollback.