Repair, Backup, Restore, and Anti-Entropy
This domain tests anti-entropy and data protection: scheduling repair inside gc_grace_seconds, taking consistent snapshots, and restoring without corrupting topology.
Repair
- nodetool repair builds Merkle trees for token ranges and streams mismatched data. A full repair covers all ranges; -pr repairs only the primary range; -local or --in-local-dc restricts scope. Run repair on each node, staggered, and complete a full cycle before gc_grace_seconds, default 864000 seconds or 10 days, to prevent zombie data resurrection.
- Incremental repair, if enabled, splits repaired and unrepaired SSTables and has operational constraints. Reaper or similar schedulers are common for large clusters.
- Hinted handoff is short-term; read repair is probabilistic or blocking depending on consistency; neither replaces scheduled repair.
Backup and restore
- nodetool snapshot -t tag creates hard links to SSTables. It is online and cheap, but the snapshot does not include the commitlog. For point-in-time recovery, also archive commitlogs or quiesce writes.
- nodetool clearsnapshot removes snapshot references; do not assume snapshots are independent full copies.
- Restore: stop the node, clear the relevant data directories, copy snapshot SSTables into the table directory while respecting keyspace and table names, restore commitlogs if needed, then start. Alternatively use sstableloader to load into a live cluster. Restore into compatible topology and run repair.
Worked example
A node dies permanently with RF=3. Do not restore that node's snapshot into a different token range. Replace it with a new node using the same tokens via replace_address_first_boot so it streams replicas from peers, then run repair. If the whole cluster is lost, restore all nodes' snapshots, then repair.
Traps
Treating snapshots as backups without commitlogs; running repair outside gc_grace_seconds; repairing all nodes at once; restoring with mismatched cluster name or topology; and assuming nodetool repair backs up data.