Apply Lag and a Regional RTO Gap on Exadata Database Service
A logistics company runs a quarter-rack X9M Exadata Database Service on Dedicated Infrastructure in the London region. The production VM cluster has two database nodes running Oracle RAC 19c, and redo ships to a single-node standby VM cluster in the same region but a different availability domain. Nightly ETL between 01:00 and 03:00 generates about 900 GB of redo; the standby's apply lag, normally under 30 seconds, climbs past 45 minutes in that window, and the standby nodes have 30 percent fewer cores than production. The nightly Level 0 RMAN backup also starts at 01:00 from the primary and writes to Object Storage over the client network, taking three hours. The business requires an RTO of 15 minutes for a regional failure and no more than 5 minutes of data loss. A quarterly Grid Infrastructure and database patching is due, and the previous cycle caused a 22-minute outage because the DBAs patched the primary first.
1. Diagnose the 45-minute apply lag and prescribe fixes in priority order.
Read the worked answer
Separate transport lag from apply lag before changing anything; they have different causes. Query V$DATAGUARD_STATS: a high transport lag points at the network, a high apply lag with low transport lag points at the standby's capacity.
Likely contributors here:
- Network contention. 900 GB of redo in two hours is roughly 125 MB/s sustained, which is a large fraction of the client network. The Level 0 backup starting at the same time also ships over the client network to Object Storage, so the two compete. Redo transport between sites uses the client network, not the RDMA interconnect, so the Exadata fabric does not help.
- Undersized standby. Thirty percent fewer cores means less capacity for parallel managed recovery.
- Standby redo log sizing. If standby redo logs are smaller or fewer than the primary's online redo logs, apply waits on log switches.
Priority order: move the Level 0 backup out of the ETL window or take it on the standby so it never competes with redo transport; size the standby to match production; verify standby redo log groups and enable adequate parallel recovery; if bandwidth remains the constraint, enable redo transport compression or consider Far Sync. Alert on both lag metrics separately so you know which one is firing.
2. Plan the patching so it does not repeat the 22-minute outage.
Read the worked answer
The 22-minute outage was almost certainly client failover time, not patch time. RAC protects against node failure but not against a role change: when you switch over, the connection endpoint changes and every session must reconnect.
Correct sequence: patch the standby first because it serves no traffic, run a Data Guard switchover through the broker so the standby becomes primary, patch the former primary, then switch back in a later window if you want the original topology. Schedule the switchover outside peak and budget it as downtime, because it is.
Then attack the reconnect time directly: the application connect descriptor must include FAILOVER=ON with RETRY_COUNT and RETRY_DELAY, an appropriate TRANSPORT_CONNECT_TIMEOUT, and the SCAN address rather than a node VIP. Add Fast-Start Failover with an observer for automation. Rehearse a switchover in a non-production pair and measure how long the application actually takes to recover, because the 15-minute RTO will be consumed by this step long before the patching itself matters.
3. Does the current topology meet the 15-minute RTO for a regional failure? Justify your answer and give options.
Read the worked answer
No. The standby is in the same region, in a different availability domain. That protects you against an AD failure and against patching, but a regional failure takes out both sites at once, and the RTO for that scenario is effectively the time to restore from Object Storage backups into a new region, which is hours, not 15 minutes.
Options, in order of preference:
- Add a cross-region standby, either directly from the primary or cascaded from the same-region standby to avoid doubling the redo shipped over the WAN. RPO stays inside 5 minutes because async replication typically lags by seconds; RTO depends on automation, so pair it with Fast-Start Failover and an observer in a third location, plus a tested DNS or connection-string switch.
- Move the standby entirely to the second region and accept that AD-level failures now require a longer recovery. This is cheaper but weaker.
- Keep the same-region standby for AD and patching events and add cross-region backups only. This does not meet the 15-minute RTO and should be rejected explicitly.
Whatever you choose, also confirm the cross-region standby has production-equivalent shape, or a failover leaves you running the business on a fraction of the capacity.