Data Guard Failover After Primary Availability Domain Loss
A production Oracle DB system runs on a 2-node RAC VM.Standard3.Flex DB system in the prod-db compartment in Frankfurt. A Data Guard standby DB system is in another availability domain in the same region, using MaxPerformance protection mode. Backups are configured to Object Storage with 30-day retention. The application connects through the primary DB system's SCAN listener; connection strings are stored in a config file on 20 application servers. At 02:14, the primary availability domain loses power. The primary DB system is unreachable. The standby shows an apply lag of 18 seconds and its Data Guard association state is Available. The incident commander must restore service within a 30-minute RTO and an RPO of 5 minutes. The team has no recent failover drill. A junior DBA suggests opening the standby read-write and repointing the application.
1. Should the team perform a switchover or failover? What checks and sequence should they follow?
Read the worked answer
Perform a failover, not a switchover. A switchover is a planned role reversal that requires the primary to be available and synchronized; the primary here is unreachable. Before failover, verify that the apply lag is within the 5-minute RPO (18 seconds is acceptable), that the association state is healthy, and that the protection mode is understood. Then use the OCI Console or CLI Data Guard failover action so the broker updates roles and metadata. Do not simply open the standby read-write and repoint clients: that bypasses broker role transitions and raises split-brain risk when the old primary returns. After failover, update the application connection strings to the new primary's SCAN listener, validate read-write transactions, and confirm backups are enabled on the new primary.
2. What are the backup and recoverability implications after failover, and what should the team do immediately?
Read the worked answer
Backups are associated with the DB system, and the standby may not have the same recent backup history as the failed primary. After failover, enable or reconfigure automatic backups on the new primary immediately and take a fresh full backup as soon as practical. The old primary's backups remain in Object Storage and can still be used for restore, but they represent the old primary's timeline. Confirm the retention policy is 30 days and that the Object Storage bucket is accessible from the new primary's compartment. Because the team has not drilled failover, also run a restore validation against a non-production DB system to prove the backups are usable, and document the new primary's backup schedule in the runbook.
3. How should the old primary be reintegrated when its availability domain recovers?
Read the worked answer
Do not open the old primary read-write. After the availability domain recovers, use the Data Guard reinstate operation to convert it into a standby of the new primary. Depending on its state, reinstate may require Flashback Database or an RMAN duplicate; the broker will indicate whether the old primary can be reinstated automatically. Verify that the old primary is applying redo and that the apply lag is acceptable before considering a switchback. If a switchback is desired, perform a planned switchover during a maintenance window, not an unplanned failover. Keep the old primary isolated from application traffic until reintegration is complete.