Migrating 210 VMs and 30 TB of NFS over a 200 Mbps Direct Connect

Usable throughput: 200 Mbps × 0.65 ≈ 130 Mbps ≈ 16 MB/s sustained.

VM disks: 38 TB ≈ 38 × 1,048,576 MB ≈ 39.8 million MB ÷ 16 MB/s ≈ 2.49 million s ≈ 29 days of continuous transfer.

NFS: 30 TB ≈ 31.5 million MB ÷ 16 MB/s ≈ 1.97 million s ≈ 23 days continuous.

Combined: roughly 50 days of a fully saturated circuit, with zero headroom and no tolerance for a restart after an interruption.

Implications:

1. A single-pass cutover is impossible; the transfer must be decoupled from the cutover. Use continuous block-level replication (SMS) so the bulk copy runs for weeks while production continues, and only the delta is applied inside the 4-hour window.
2. A saturated circuit is a business risk, not just a schedule risk. Throttle replication during business hours and reserve full bandwidth for windows - this is precisely what failed last time.
3. The 30 TB NFS is cold, large-object engineering data with no daily change. Streaming it over the circuit is the wrong tool; use an offline transfer appliance (DES) and reconcile deltas over the circuit afterwards.
4. Oracle is small (6 TB) and change-heavy - it belongs on DRS or native replication, never on block-level sync.

The judgement: bandwidth math, not tooling, sets the migration architecture.

Wave 0 - landing zone (weeks 1-2). VPCs and subnets without overlapping the on-premises ranges, security groups, IAM and enterprise projects, logging, CBR policies, a second Direct Connect or VPN for redundancy, and a DNS TTL reduction scheduled one week before each cutover.

Wave 1 - stateless web/app tiers. SMS continuous sync, test-start the ECS instances in the cloud against a read-only database, then a 4-hour Saturday cutover with DNS switch. Lowest risk, validates the whole pipeline end to end.

Wave 2 - Oracle databases. Use DRS/native replication with a schema and stored-procedure validation pass on a restored copy first. Cut over after Wave 1 proves DNS and connectivity, because every other application depends on these.

Wave 3 - NFS engineering data. DES appliance to OBS or SFS Turbo for the 30 TB bulk, then a delta reconciliation. This is sequenced late because it is the longest-running and least reversible step, and running it in parallel with database work would contend for the same circuit.

Wave 4 - decommission and cost review. Confirm backups exist in the cloud, run the DR drill, then release on-premises capacity.

Ordering rationale: each wave reduces uncertainty for the next (connectivity → stateless → stateful → bulk), and each cutover window is small enough to roll back within the Saturday night.

1. Crash-consistent, not application-consistent, database copies. Block-level replication of a running Oracle instance yields a recoverable-but-inconsistent image. Control: use database-native replication for Oracle; for anything else, quiesce or use VSS-consistent snapshots, and test-start and validate before every cutover.
2. Circuit contention with production ERP. Control: bandwidth caps on replication jobs during 08:00-18:00, a hard change-freeze on bulk starts, and CES alarms on circuit utilisation above 70% during business hours.
3. IP and DNS failures at cutover. Control: re-IP plan validated in Wave 1, TTL lowered in advance, and a tested reverse switch for rollback.
4. Licensing and driver compatibility. Windows and Oracle BYOL terms, plus SMS agent and driver support on obsolete guest OS versions - validate on a sample VM per OS release before committing the wave.
5. Thick-to-thin capacity planning. 38 TB of thick disks will not map one-to-one onto cloud pricing expectations; size from actual used blocks and confirm with performance metrics, not provisioned capacity.
6. No rehearsed rollback. Control: each wave gets a documented 30-minute rollback decision point inside the 4-hour window; if the decision point passes without green checks, revert.
7. Knowledge transfer gap. Control: the on-premises operators shadow the first two cutovers and the runbook is signed off before Wave 2.