The DR Drill That Lost 3,100 Orders: WebLogic JMS and Data Guard Together

The file store is the problem. A WebLogic file store lives on the local disk of the instance that owns it. When the database fails over to Phoenix, nothing about the Ashburn instances changes except that they can no longer reach their primary database - the 18 GB of undelivered messages sit on disks in a region you are evacuating, and any recovery that starts the domain on new instances starts with an empty store. Messages that had been accepted from producers but not yet consumed are simply gone. That is a store design failure, not a Data Guard failure.

What to change:

1. Move persistent messaging to a JDBC store in the database. The store then participates in Data Guard replication, so messages accepted before the failover exist on the standby when it is promoted. This is the single highest-value change and it directly addresses loss.
2. Get the ownership rules right. A JDBC store must have exactly one writer. During failover you must fence the old domain - stop its managed servers, or withhold its database connectivity - before the new domain starts, or two domains will write the same store tables and corrupt the schema and the ordering. Sequence the runbook: fence, promote, start, verify.
3. Size for backlog. 18 GB of undelivered messages at failover means the consumer tier was already behind. A DR site that inherits a 20 GB backlog cannot meet a 30 minute RTO even with perfect replication. Alert on store depth and drain rate, and set a threshold that blocks new orders gracefully rather than silently accumulating.
4. Consider store-and-forward for cross-site handoff so the producer side can buffer locally and forward once the target is healthy, rather than depending on a single remote store being available during the transition.

Two independent defects: discovery and reconnection.

The static IP in the TNS alias guarantees a stale target. After a role transition the old primary's address either does not answer or answers as a broken instance. Every reconnect attempt burns a TCP timeout, pools fill with dead connections, and the domain appears hung.

Redesign:

1. Use a role-aware connection descriptor with multiple addresses and the Data Guard broker's services, so the alias resolves to addresses that follow the role rather than a frozen IP. Lookups must be able to fail over across addresses.
2. Use a GridLink data source with FAN/ONS. Pools then receive an event when the role changes and evict connections immediately, instead of discovering the failure request by request. This alone typically turns tens of minutes of thrash into seconds.
3. Enable Application Continuity with replay for the transaction path, so in-flight work that hits a role transition is re-executed rather than returned as an error to the customer. Verify that the application's transactions are replay-safe: no non-idempotent side effects outside the database.
4. Bound the failure cost. Set connect and read timeouts that are shorter than the RTO budget, so a dead endpoint is abandoned quickly instead of consuming the whole window.
5. Fix the ordering of recovery. WebLogic will not start cleanly until the database is accepting connections on the new primary; the runbook must gate domain startup on a successful database readiness probe, then start servers, then open traffic, then confirm queue consumers.

Measure the result end to end: from last successful order to first successful order, not from broker command to database open.

JMS over a WAN between two sites, and any store that survives a role transition, gives at-least-once delivery. When a consumer's acknowledgement is lost - because the consumer died, the connection dropped, or the store was reopened during failover - the message is redelivered. Duplicates are the expected behaviour of the guarantee, not a bug. Exactly-once is not available across this boundary; it has to be constructed.

Design:

1. Idempotent consumer keyed on a business identifier. Every order carries an order ID assigned by the producer before the message is sent. The consumer writes the business effect and the dedup record in the same local transaction, so either both exist or neither does. A redelivered message hits the unique constraint and is acknowledged and discarded.
2. Include the JMS message ID and a producer instance identifier in the dedup key, so that two genuinely distinct orders are never collapsed by a reused business key.
3. Preserve per-order ordering with a unit-of-order keyed on order ID. Redelivery out of order is a different corruption from duplication and harder to detect downstream.
4. Control redelivery. Set a bounded redelivery limit, route poison messages to an error destination with the original payload and headers, and alarm on that destination. Unlimited redelivery hides the defect and inflates the backlog.
5. Make the duplicate visible. Emit a metric on dedup hits, because a rising rate signals failover instability or consumer timeouts long before anyone sees a wrong order.

Finally, decide explicitly what the Azure consumer does with a duplicate that arrives after the business window closed - reject, ignore, or reconcile - and encode that decision in the dedup table with a retention period longer than the maximum redelivery window.