12 TB Into a 6-Hour Hole
A team must move a 12 TB analytics warehouse from a legacy platform to a new managed service. The migration window is a single 6-hour maintenance slot on a Saturday. Compliance requires that all data, including backups and temporary files, remain inside the country; the new service's default configuration sends snapshots to a multi-region bucket. Downstream dashboards query the warehouse continuously and tolerate at most 15 minutes of read unavailability. A profile of the source finds that roughly 3% of rows carry schema variations introduced by an older ingestion client: two columns sometimes arrive as text instead of numeric. The team has one rehearsal slot before cutover.
1. Choose the migration strategy for the 6-hour window and justify it against 12 TB and a 15-minute read tolerance.
Read the worked answer
A single 6-hour bulk copy is not a plan; the arithmetic rules it out.
12 TB in 21,600 s requires about 555 MB/s sustained end-to-end - read, transform, write, and index build - with zero slack for validation, retries, or a slow link. Any single degradation consumes the window.
Instead, split the copy across the window's edge:
- Bulk-copy the bulk ahead of time while the source keeps serving, from a consistent snapshot anchored at a known point (LSN or timestamp).
- In the window, replay only the change stream from that anchor and cut over. Size the delta explicitly: at ~2% daily change, that is ~240 GB, which at 100 MB/s is about 40 minutes, leaving real margin for validation and the flip.
- Keep both systems readable behind a routing flag. The 15-minute tolerance means you can validate against the new warehouse while dashboards still read the old one, and flip per-dashboard. The only moment of write unavailability is the cutover itself, which should be seconds of routing plus the final delta.
Use the rehearsal slot for the cutover, not the bulk load. The bulk copy and the delta-replay tooling should already be proven by then. Spend the rehearsal on the routing flip, the validation suite, and the rollback path - those are what the Saturday depends on.
2. How do you handle the 3% schema drift, and what validation would you require before cutover?
Read the worked answer
Profile the 3% before choosing a treatment. Which of the two columns, how often, and - critically - whether the text values are parseable numbers or genuinely non-numeric (empty, 'N/A', locale-formatted '1.234,56'). The three cases need three different answers.
- Parseable values: explicit, logged cast in the loader, with failures routed to a quarantine table rather than dropped.
- Non-parseable values: decide per column with the data owner - null plus a quality flag, a sentinel, or a string-typed shadow column. Document it in the schema contract so it is a decision, not an accident.
- Load into a strict-typed staging table, never the final table. A warehouse that casts 'N/A' to 0 does not fail; it silently poisons every average and total built on that column. A failed load is cheap; a wrong dashboard that people trust is not.
Validation gate before cutover:
- Row counts per table and per partition (partitions catch what totals hide).
- Checksums or sums on a fixed set of numeric columns.
- Around 20 golden queries with answers pinned from the legacy system, agreed with the dashboard owners - not just with engineering.
- Null-rate comparison per column against the source profile.
- Quarantine count reconciled against the profiled 3%. A number far below it means rows vanished, not that the data was clean.
3. Compliance says data must stay in-country, but the new service defaults to multi-region snapshots. What do you do, and what is your rollback trigger?
Read the worked answer
Treat region pinning as a blocking design item with evidence, not a post-migration checklist line.
- Disable multi-region snapshot replication at the service level, then prove it: write a uniquely named canary object, trigger a snapshot, and confirm no replica appears in the out-of-country bucket.
- Follow the temporary path too. Many services stage load files and query exports in a default bucket that follows a different region setting than primary storage. Compliance wording usually covers temporary files and backups explicitly - this scenario's does.
- Check backup retention separately from snapshot replication; they are frequently distinct settings.
- Retain the evidence - config export plus canary result - in the migration record. A vendor assurance or an architecture diagram is not evidence.
Rollback trigger, defined before the window opens:
- Any golden-query mismatch beyond the agreed tolerance.
- Quarantine count above 1.5× the profiled 3%.
- Read unavailability exceeding 15 minutes.
- Any observation of cross-region replication.
Make it actionable: keep the source live and read-only after cutover, and keep the routing flag reversible for at least 7 days. Evaluate the trigger within the first two hours of the window - a trigger you can only assess after the window closes is a post-mortem, not a rollback.