The Cutover Weekend with Rising Replication Lag
A healthcare SaaS provider is cutting over 14 applications and 60 VMs from its primary data center to a cloud region over a single weekend. The runbook schedules DNS changes at Sunday 02:00 with a 3,600-second TTL. At Friday 18:00 the database replication lag is 12 minutes and rising because a reporting job is running. Three applications have never had a rollback test, and two interfaces exchange files with a partner over SFTP every 15 minutes. The business has a 04:00 Monday service-level deadline for claims intake. The cutover team has 18 people across three time zones. The change advisory board has approved the window but not a rollback decision authority. A previous cutover at another site failed because the application connection strings were cached in a middleware layer.
1. Define the gating criteria that must be met before any DNS change at 02:00.
Read the worked answer
Do not change DNS at 02:00 with a 3,600-second TTL and a 12-minute replication lag. Gates: stop the reporting job and confirm replication lag below two minutes for at least 60 consecutive minutes; lower the DNS TTL to 60 seconds at least 48 hours before cutover; flush middleware and application caches; confirm all connection strings are externalized; test partner SFTP connectivity from the target; and have rollback tested for all 14 applications, not 11. The three untested rollbacks are a hard stop for those apps; either test them or exclude them from the wave. At 02:00, verify data sync, freeze writes, run final validation, then switch DNS in stages. Because old TTL records can persist for an hour, keep the source running and reachable. Staff the bridge with named owners for database, network, applications, and partner interfaces.
2. Establish rollback authority and decision points for the cutover, given the advisory board's gap.
Read the worked answer
Rollback authority must be explicit before the window. Assign one incident commander and one business owner who jointly own the decision; the change advisory board approves the window, not the in-flight call. Set decision points: 02:30 data validation, 03:30 application smoke tests, 04:00 partner file exchange, 05:00 claims intake readiness. Pre-agree abort triggers: replication lag not converging, more than two critical applications failing smoke tests, partner SFTP failure, or claims intake not processing by 05:00. If aborted, revert DNS, flush caches, restart middleware, and resume source writes; verify no duplicate claims. If committed, keep the source read-only for 72 hours. The previous failure with cached connection strings is a warning: add a cache-flush and restart step to both the forward and rollback runbooks, and test it.
3. Describe the post-cutover validation and communication needed before declaring success on Monday.
Read the worked answer
Validate end-to-end business transactions, not just server pings. For claims intake, submit a synthetic claim through every channel, confirm adjudication, database write, and downstream file. For the two SFTP interfaces, exchange a test file in both directions and verify timestamps and sequence numbers. Check row counts and replication reconciliation for the databases, DNS resolution from each application subnet, authentication, and print or output paths. Monitor for 72 hours with heightened thresholds. Communicate status at each decision point to the business and partner. After cutover, keep the rollback option warm until the Monday 04:00 deadline passes and the first real claims batch completes. Record actual timings against the runbook and convert gaps into remediation items. A cutover is not complete when DNS changes; it is complete when the business service is proven.