Cassandra K8s: Backup, restore, and disaster recovery

This domain tests whether data can be recovered after node loss, namespace deletion, or a bad upgrade, not merely whether a snapshot command exits zero.

Core rules

• A local Cassandra snapshot is not a disaster-recovery backup; it lives on the same volume.
• Medusa backs up SSTables and schema to object storage such as S3 and supports restore into a cluster.
• CSI volume snapshots capture volumes but need coordination so writes are quiesced or consistent.
• Define RPO and RTO: backup frequency sets RPO, restore procedure and capacity set RTO.
• A restore should be tested into an isolated namespace and validated with row counts, schema, and repair.

Worked example

A team takes nightly Medusa backups to S3 and also keeps local snapshots. The namespace is deleted by mistake. Local snapshots disappear with the PVCs, so only the S3 backups matter. Recovery creates a new datacenter with matching keyspace replication, restores the backup, then runs repair and validation queries. Because the last backup was 20 hours old, the RPO is 20 hours, not nightly in theory. If the restore target has a different node count, token ownership must be handled by the restore tooling or a fresh cluster.

Traps

• Treating nodetool snapshot alone as off-cluster backup.
• Never testing a restore.
• Backing up during a topology change without verifying success.
• Assuming a backup includes monitoring, secrets, and operator configuration.