Order Management Cutover: 340 Interfaces, Partner IP Whitelists, and a Six-Hour Window

The rehearsals have already told you where the risk is; the runbook should be rebuilt around that evidence rather than around the standard sequence.

Fix the identified problem first. Both overruns were interface reconnection. That means the interface step must move from a serial tail-end activity to a pre-staged, parallelised one: pre-validate connectivity and credentials for all 340 interfaces days before cutover, pre-load configurations, and run interface reconnection as parallel batches with named owners per batch, not one engineer walking a list. Rehearse the interface step alone, repeatedly, until it fits in 45 minutes.

Sequence:

1. T-48h: lower DNS TTL to 60s, confirm partner whitelists and new egress IPs are in place (this is a lead-time item - 22 partners will not all respond quickly), freeze changes.
2. T-0: stop application, quiesce batch, confirm Data Guard/standby sync position, take final backup.
3. Switch database and application, then interfaces in parallel batches.
4. Validate, then repoint DNS and monitor.

Go/no-go gates at defined times: database sync confirmed; application smoke tests green; at least X% of interfaces confirmed connected; batch scheduler validated. If a gate fails at its deadline, stop.

Rollback triggers, agreed in advance and owned by the business, not the DBA:

• Interface connectivity not achieved by T+3h (the recurring failure mode).
• Data validation mismatches above an agreed tolerance.
• Contact centre unable to take orders for more than 30 minutes.

Rollback must be rehearsed, not assumed. With DNS at 60s TTL, rollback is roughly the reverse, but only if the on-premises environment was kept warm and write-synced. If it was powered down at T-0, you have no rollback - you have a hope. Decide explicitly which it is and state it in the runbook.

Validation ordered by 'cheapest and most reversible first, most business-critical last' - you want to fail fast and in a place where failure is inexpensive to fix.

1. Infrastructure and data integrity (first 60 minutes). Row counts and checksums per critical table, sequence values ahead of the maximum used key, index and constraint validity, invalid objects, and the standby's apply lag against the 15-minute RPO. Any mismatch here is a stop, not a warning.

2. Application smoke (next 60 minutes). Login, order creation, price lookup, inventory decrement, payment authorisation against a test card - the shortest path that proves the whole stack responds. Then batch scheduler startup and the first few jobs, because 1,900 jobs failing at 02:00 is the classic post-cutover incident.

3. Interface validation. For each of the 340, a defined proof: successful connection, a handshake/message round trip, and for the 22 whitelisting partners, an explicit confirmation that traffic from the new egress IP is accepted. Whitelisting confirmation is a business validation, not a technical one, and needs a named owner per partner.

4. Business validation. A reconciliation of order totals and open orders against the pre-cutover snapshot, executed by finance or operations, not by IT. Sign-off comes from the business owner.

What makes it actionable: each check needs a defined expected result, a named owner, a timestamp, and a threshold that triggers either continue or rollback. 'Ping responds' is not a validation - the current single ping check per server would have passed while every interface was down. Capture evidence automatically into one runbook sheet, because the post-incident review will ask what was checked and by whom, and memory will not survive the week.

Cutovers rarely fail on day one; they fail in week three, when the war room has disbanded and nobody is watching.

Sizing from 11% averages is the biggest latent risk. Average utilisation hides the peaks that matter: month-end batch, promotion days, and the contact centre's morning burst. Expect shapes that are simultaneously too big on average and too small at peak. Plan a rightsizing review at day 14 after collecting real metrics, using p95 and peak, not mean - and check the BYOL implications, because shrinking a shape is a licensing decision as much as a performance one, and on dedicated shapes you can only reduce cores to what the licence entitles.

Observability is the second gap. One ping per server is not monitoring. Before day 1 you need: database and host metrics, WebLogic JVM and thread-pool health, synthetic order-placement transactions end to end, interface queue depth and failure counts, and batch run duration versus baseline with alerting on overrun. Then alert on symptoms the business cares about (order failure rate, batch overrun) rather than only on infrastructure. Set the alert thresholds from the rehearsals and the first week of real traffic, not from guesses.

Other predictable week-1 to week-4 items:

• Cost spike. Untagged or over-provisioned resources and forgotten DR assets inflate the first bill; verify tag enforcement is working and review the cost report against the forecast weekly for the first month.
• Backup and restore confidence. A successful backup is not a successful restore. Run a restore test inside the first two weeks - the RTO commitment is unproven until you do.
• Job scheduling drift. Batch runtimes shift on new hardware; re-baseline and adjust dependencies.
• Runbook and access handover. Confirm the operations team - not the migration team - holds the production access, and that the break-glass procedure has been exercised at least once.
• Retire the source. Decommissioning is a scheduled task with a date, or the on-premises estate will still be running (and billing) in month six.