CBS Batch Restart After Asynchronous DR Failover
A regional rural bank runs its core banking system on a two-node Oracle RAC primary at the data centre and an asynchronous logical standby at a disaster-recovery site 180 km away. The nightly interest-accrual batch runs from 01:00 to 03:30 and posts roughly 9 lakh entries. During a storage-controller failure at 02:10, the primary node restarts, the batch job is restarted by the scheduler, and the bank fails over to the standby at 02:40. The next morning, reconciliation finds 42 accounts with duplicate interest postings and 6 accounts with no accrual for the day. DR replication lag at the time was 8 minutes; the bank's stated RPO is 15 minutes and RTO is 2 hours. The application team proposes switching to synchronous replication and running all batches only on the DR node. The IT head asks for a technical review before the next month-end.
1. Diagnose the most likely cause of the duplicate and missing postings. Does the observed lag alone prove an RPO breach? Explain.
Read the worked answer
The lag figure of 8 minutes is inside the 15-minute RPO, so it does not by itself prove an RPO breach. RPO measures tolerated data loss, not application-level consistency. The more likely cause is that the batch was restarted without an idempotent checkpoint. Some accounts had already been posted before 02:10; when the scheduler reran the whole job, those accounts were posted again. The six missing accruals are consistent with in-flight transactions that were neither committed on the primary nor available on the standby at 02:40, or with a control table that marked the batch complete before the final commit.
The correct diagnosis therefore separates replication lag from batch transaction integrity. Reconciliation should compare the batch run ID, account-level audit trail, control totals and the DR redo position. A duplicate is a restart-control failure; a missing accrual is either a genuine data-loss event or a partial-rollback event. The bank should not simply tighten RPO until it proves which of these occurred.
2. Evaluate the proposal to use synchronous replication and run all batches only on the DR node. What conditions must be satisfied before accepting it?
Read the worked answer
Synchronous replication would reduce committed-transaction loss, but it is not a complete fix. Every commit waits for the standby, so the added round-trip latency can slow batch throughput and may affect online transaction response times. At 180 km, the network round trip is not the only variable; storage acknowledgement, bandwidth and peak redo volume matter. The bank must benchmark the 9 lakh-entry batch under synchronous mode and confirm that the 01:00 to 03:30 window is still achievable.
Running batches only on the DR node also changes the operating model. The DR node needs equivalent CPU, memory, storage and backup capacity, plus clear role-reversal and failback procedures. Split-brain protection and quorum rules must be tested. Most importantly, synchronous replication does not make the batch idempotent: a restart can still double-post unless the application uses a unique run key and checkpoint table. Accept the proposal only if latency tests pass, DR capacity is proven, and batch restart logic is fixed independently.
3. List the immediate controls you would apply before rerunning the accrual batch.
Read the worked answer
First, disable the automatic scheduler and freeze the affected run ID so that no further retries occur. Second, extract the account-level audit trail and compare primary and standby control totals to identify every duplicate and missing entry. Third, do not reverse duplicates by ad hoc SQL; use a controlled reversal run with maker-checker approval and a unique key on account, product, value date and run ID. Fourth, insert an idempotency check so that a rerun skips accounts already posted successfully. Fifth, run the corrected batch in shadow mode on a restored copy and reconcile the total interest to the general ledger. Only then should the bank obtain business sign-off, rerun the batch and communicate the correction to affected customers.