Cassandra K8s: Scaling, upgrades, and rolling operations

This domain tests change control: adding or removing nodes and changing versions without dropping below quorum.

Core rules

• Scale through the CassandraDatacenter size; the operator adds nodes and bootstraps them.
• Scale down requires decommissioning so token ranges move before the node leaves.
• Upgrade the operator and CRDs before changing the Cassandra server version.
• Rolling upgrades replace one node at a time and wait for readiness; a PodDisruptionBudget must not block drains.
• Take a backup and verify cluster health before any version change.

Worked example

A six-node cluster with RF 3 and QUORUM is upgraded from one minor version to the next. The operator changes serverVersion, then replaces one Pod, waits for it to rejoin and become ready, and proceeds. With replicas spread across three racks, one unavailable node leaves two replicas per partition, so quorum is maintained. If an administrator instead changes size and serverVersion in one edit, the rollout becomes harder to reason about; separate the changes, watch status, and confirm nodetool status shows all nodes UN before continuing.

Traps

• Scaling down without decommission, losing ranges and data.
• Reducing below the keyspace replication factor.
• Skipping a required intermediate major version.
• Setting a PDB that prevents voluntary eviction during the upgrade.