Table-Level Restore From Medusa Into a Smaller Cluster in Another Region
Medusa runs nightly full backups and 6-hourly incrementals to S3 for a 12-node, 3-rack cluster with RF=3 and 4 TB per node. At 14:05 an engineer runs a migration that drops and recreates the orders table with a changed primary key; the old data is gone. The last full completed at 02:00, and an incremental completed at 12:00; the 18:00 incremental has not run. You must recover the orders table as of 13:55 into a new 6-node cluster in another region. The new cluster has a different cluster name, a different number of nodes, and the same Cassandra major version. The business wants read traffic on the recovered table within four hours. Backups store the schema and token metadata; the restore tooling can restore a whole cluster or selected keyspaces.
1. Which restore target and scope do you choose, and what must be true before the 02:00 full plus the 12:00 incremental can produce a 13:55 recovery point?
Read the worked answer
The correct scope is the orders keyspace or table, not the whole 48 TB cluster; a full restore to 6 nodes cannot finish in four hours. But the requested 13:55 recovery point is not achievable from the stated backups. A full at 02:00 plus an incremental at 12:00 gives a restore point at 12:00, not 13:55. Medusa incrementals are not continuous commitlog archiving. Unless commitlog backups or a later incremental exist elsewhere, the honest answer is to recover to 12:00 and accept a 1 hour 55 minute RPO, or to recover to 13:55 only if the backup catalog contains commitlog or point-in-time data that was not mentioned.
I would verify the backup catalog first, then restore the keyspace into a temporary cluster, validate it, and use sstableloader to load only the orders table into the target. The 02:00 full and 12:00 incremental are both required; restoring only the incremental is invalid.
2. What topology and metadata mismatches matter when restoring a 12-node backup into 6 nodes, and how do you handle token ownership?
Read the worked answer
The source ring has 12 nodes and 16 vnodes per node; the target has 6 nodes and will generate a different token map. If the restore tool writes each old node's SSTables onto a corresponding new node, the data will be on nodes that do not own those ranges in the new ring. Coordinators route queries by the new ownership, so rows can appear missing until repair or rebuild streams them. The partitioner and num_tokens must match, and the schema must be the pre-migration schema for orders; otherwise the SSTables may not load or may load with the wrong types.
Two workable approaches: restore into a temporary 12-node cluster with the same num_tokens and token assignments, verify, then use sstableloader from that cluster to the 6-node target; or restore the SSTables to a staging area and sstableloader them directly into the target, letting the target's ring place the data. Do not copy old data directories onto the 6 nodes and start Cassandra. Also confirm RF=3 still fits the target's 3 racks and that the restore configuration handles the different cluster name.
3. Give a validation and cutover checklist for the `orders` table.
Read the worked answer
Start with schema: confirm the orders table definition, primary key, secondary indexes, materialized views, and TTL settings match the pre-13:55 state, not the broken migration. Then validate data: compare row counts for known partitions, run checksum queries on a sample of partitions, and check min and max timestamps to confirm the 12:00 cutoff. Run nodetool tablestats and nodetool verify on the restored SSTables, then nodetool repair on the target so all replicas agree.
Test the actual order-lookup and reporting queries against the restored table, including paging and consistency level. Cut over by pointing the application at the new table or cluster with a feature flag, keep the old cluster read-only, and monitor read latency and error rates. Document the RPO gap and fix the backup schedule: add commitlog archiving if 13:55-style recovery is required, and test a table-level restore before the next migration.