A public Autonomous Database, a 15-minute RPO, and a cost spike nobody can explain
Fairway Grocers runs an Autonomous Database (Transaction Processing, Serverless) on a public endpoint in the Frankfurt region for store operations, plus a two-node RAC DB system on Exadata for the legacy warehouse. 200 stores connect to the ADB over an IPSec VPN terminated at a CPE that is attached to a DRG in the VCN. The security review has three findings. First, the ADB endpoint must not be reachable from the internet. Second, the RPO for a full region outage is 15 minutes and the RTO is 1 hour, but the only protection today is a nightly RMAN backup to a local bucket. Third, nobody can answer "is the standby behind?" or "why did database spend jump 30% last month?". The ADB is sized at 8 ECPUs with auto-scaling enabled and averages 70% CPU. Traffic has been flat for six months.
1. Describe the target private-endpoint design for the ADB and every network and DNS dependency that has to line up for the stores to keep working.
Read the worked answer
Target state. The ADB sits on a private endpoint in a dedicated private subnet, for example 10.40.8.0/24, with no internet gateway on that subnet and no public IP anywhere in the path.
Dependencies, in order:
- Endpoint placement. The private endpoint consumes an IP in the chosen subnet. Use a dedicated subnet so that security rules and route changes for the database never accidentally affect application servers.
- Ingress rules. An NSG on the endpoint (or the subnet security list) allowing TCP 1521 from the on-premises store CIDR only - not from 0.0.0.0/0 and not from the whole VCN unless a bastion genuinely needs it.
- Return routing. The subnet's route table needs a route for the on-premises CIDR through the DRG, and the on-premises side needs a route for 10.40.8.0/24 through the tunnel. Asymmetric routing here produces connections that hang rather than fail, which is harder to debug.
- Outbound paths. The database still needs paths for service management and patching. Keep the NAT or service gateway rules on that subnet; do not strip egress in the name of security without testing.
- Name resolution. This is the usual failure. On-premises clients must resolve the ADB hostname to the private IP. Either give them a DNS alias to the private IP, or create a private DNS zone and configure the on-premises resolver to forward that zone to the VCN resolver. Test resolution from a store-network client, not from a bastion inside the VCN.
- Tooling. SQL Developer Web, OML and other console-driven tools follow the same endpoint; confirm they are still reachable by the people who use them.
Migration path. If in-place conversion from public to private is not offered for the existing instance, build a new private-endpoint ADB and move the schema with Data Pump or a clone, then repoint the stores' connection strings. Plan a cutover window; do not attempt this on a Friday.
2. The RPO is 15 minutes and the RTO is 1 hour for a full region outage. Evaluate the DR options for the RAC system and for the ADB, and state why backups alone are not the answer.
Read the worked answer
Why backups fail the requirement. A nightly RMAN backup gives an RPO of up to 24 hours, far outside 15 minutes, and an RTO measured in hours: provision a host, install the software, restore the database, recover, then reopen the service. Cross-region backup copies improve durability, not RPO. Backups are necessary; they are not a DR strategy at these targets.
RAC warehouse - Data Guard. A physical standby in a second region with continuous redo transport gives an RPO measured in seconds and an RTO in minutes, provided failover is automated. Three design points people miss: (a) automate with Fast-Start Failover and an observer, otherwise the RTO includes however long it takes a human to wake up and decide; (b) the DR region needs its own VCN with a non-overlapping CIDR, its own DRG and its own security rules - a standby you cannot reach is not a standby; (c) if redo travels over the VPN rather than a second FastConnect, size and encrypt that path, because redo volume at peak is unforgiving.
ADB - Autonomous Data Guard. Enable a standby in a second region. Failover is a console or CLI operation taking minutes, which meets a 1-hour RTO comfortably, and RPO is typically well under the 15-minute target. Keep cross-region backup copies switched on as well, because replication faithfully reproduces a logical corruption.
Both: take backups of the standby as well as the primary, and document the client-side change. If the connection strings point at one hostname, failover means editing every store. Put both endpoints in the connect descriptor (or a load-balancer or DNS name in front) so failover is invisible to clients.
Test with a switchover, not a theory. A standby that has never been switched to is a hope.
3. How would you answer "is the standby behind?" and "why did database spend jump 30%?" with OCI tooling?
Read the worked answer
Replication lag. Data Guard exposes transport lag and apply lag as database metrics in OCI Monitoring. Create an alarm on apply lag against a threshold derived from the RPO - with a 15-minute RPO, alert well before 15 minutes, not at 14 - and route it to a Notifications topic that reaches the on-call rota. Add alarms on redo transport errors, archive log gaps, and role changes, because a broken transport can show a flat lag metric right up until someone tries to fail over. For the ADB, Autonomous Data Guard reports equivalent metrics on the ADB resource. Back the metrics with a synthetic check that writes a row on the primary and reads it on the standby; that catches the case where the metric is green and the data is not moving.
Cost. ADB Serverless bills per ECPU-hour, and auto-scaling lets the instance use up to three times the base ECPU. A database averaging 70% CPU on an 8-ECPU base will burst above base routinely, and those bursts are billed. With traffic flat for six months, a 30% jump points at one of three things, in this order of likelihood:
- Sustained auto-scaling bursts. If the peak is the normal state, raising the base ECPU is cheaper than paying for bursts indefinitely.
- Storage growth - tablespaces, the recycle bin, and backup retention all bill separately from compute. Check the storage metrics before blaming compute.
- A new consumer: a reporting query, a batch job, a new store.
Do it with instruments, not anecdotes. Cost Analysis grouped by a cost-tracking tag on the ADB and the DB system, a budget per compartment with both actual and forecast alerts at sensible thresholds, and a monthly review of CPU and storage utilization. Remember that budgets alert and do not enforce; if a hard ceiling is genuinely required, use compartment quotas.