Cassandra K8s common mistakes
Cassandra K8s · Understand the traps and how to avoid them.
Treating Cassandra pods like stateless Deployment replicas: scaling with a Deployment or deleting pods freely and expecting the data to follow.
Candidates carry over stateless habits and miss that each Cassandra node owns a deterministic token range on its own disk; a rescheduled pod without its PVC is a new, empty node, and the old data is orphaned or the ring is left under-replicated.
What to do instead
Always model Cassandra as a StatefulSet with stable ordinal-based identities and one PVC per ordinal via volumeClaimTemplates. Verify with kubectl get statefulset,pvc that each pod has a bound claim, and confirm the node's tokens after any reschedule with nodetool ring before declaring the operation complete.
Setting the Cassandra rack or topology labels to arbitrary values (or leaving them all identical) so the operator places all nodes in one rack or one topology domain.
The rack name drives replica placement and the operator's failure-domain awareness; a single rack silently collapses replication factor into one physical domain, so a node or zone loss takes out multiple replicas of the same partition.
What to do instead
Map Cassandra racks to real Kubernetes topology (zones or node groups) and ensure rack labels match the failure domains you actually want to survive. After deployment, run nodetool status and check that replicas of the same keyspace are spread across racks, and keep RF <= number of racks for critical keyspaces.
Assuming the operator's default storage class is production-suitable and never checking access mode, reclaim policy, or volume binding for the data PVCs.
Default StorageClasses often use Delete reclaim policy and delayed binding; candidates test on a throwaway cluster and never notice that deleting a PVC destroys data or that pods schedule before volumes are provisioned.
What to do instead
Inspect the StorageClass used by volumeClaimTemplates: confirm volumeBindingMode, reclaimPolicy, and that access mode matches the CSI driver (ReadWriteOnce per pod is normal). For anything you care about, set a Retain policy or snapshot before destructive tests, and verify PVC-to-PV binding after scaling.
Skipping the pre-upgrade checks: changing the Cassandra version or the operator/CRD version in one step without reading the CRD schema or the compatibility matrix.
Operator CRDs and Cassandra major versions have ordering constraints; candidates apply a new manifest that changes several fields at once, the operator rejects or partially reconciles it, and the cluster ends in a mixed-version state that is hard to roll back.
What to do instead
Upgrade in explicit stages: CRDs first, then the operator, then the Cassandra version, one minor version at a time, and never skip a major. Diff the CR against the installed CRD schema (kubectl explain on the CRD) before applying, and confirm each stage reports healthy before proceeding.
Assuming a rolling restart is safe by default and triggering it without checking replication, live traffic, or readiness before moving to the next pod.
Cassandra needs enough live replicas to satisfy RF and consistency level while a node is down; candidates restart all pods or let the operator proceed without confirming the previous pod rejoined the ring, causing unavailable ranges or client timeouts.
What to do instead
Before rolling, confirm RF and consistency level tolerate one node down, check nodetool status for UN (Up/Normal) on all nodes, and verify readiness probes reflect actual gossip/token state. Roll one node at a time and wait for it to rejoin as UN before continuing; drain or pause clients if CL cannot be met.
Configuring client connectivity against pod IPs or the headless service alone, ignoring the distinction between the internal seed/gossip endpoints and the external client service.
Pod IPs change on reschedule and headless DNS returns per-pod addresses; candidates hardcode these in the app or test only from inside the cluster, then production clients fail after a restart or from outside the network.
What to do instead
Use the operator-provided client service (or a dedicated service/ingress) for application traffic and the headless/seed service only for internals. Verify with a real client from the same network path the app uses, check DNS resolution of the service from a test pod, and confirm the contact points list uses service DNS names, not IPs.
Enabling TLS or authentication on only part of the path: internode encryption on but client encryption off, or auth configured without updating client credentials and the operator's superuser secret.
Cassandra's TLS is configured per channel (internode, client, and sometimes JMX), and enabling auth changes what every client and health check must present; candidates enable one setting and then chase connection errors, or lock themselves out by not tracking the generated superuser credentials.
What to do instead
Treat internode, client, and JMX TLS as separate switches and enable each deliberately, verifying with nodetool and a client connection after each. For auth, retrieve and store the operator-managed superuser secret before enabling, then create per-application roles with least privilege and test a real client login.
Treating backup as a manifest checkbox: configuring a backup resource but never restoring it, or restoring without matching the schema and topology of the target cluster.
A backup that has never been restored is unverified; candidates also restore into a cluster with different keyspace definitions, RF, or node count and hit schema disagreement or missing ranges, which only surfaces during the drill.
What to do instead
Run restore drills on a separate cluster: create the keyspace and schema, restore the snapshot, run nodetool rebuild or repair as needed, and validate row counts and queries. Record the exact restore procedure and required target topology, and schedule periodic drills rather than trusting the backup resource alone.
Reading only pod status or the operator's Ready condition and calling the cluster healthy, without checking Cassandra-level metrics and repair/compaction state.
Kubernetes readiness reflects the operator's view, not gossip, pending compactions, hinted handoff, or repair backlog; candidates miss that a node is up but not serving its ranges or that a repair has not completed in weeks.
What to do instead
Combine layers: kubectl conditions for lifecycle, nodetool status and tpstats for ring and thread health, and Prometheus metrics (pending compactions, hinted handoff, repair progress, read/write latency) for trends. Set alerts on repair age and compaction backlog, and confirm the metrics endpoint is actually scraped.
Decommissioning or removing a node by deleting its pod/PVC instead of using the operator's scale-down or decommission path.
Cassandra needs to stream its ranges to remaining replicas before leaving; a hard delete leaves the ring with unreplicated ranges and, if the PVC is also removed, no way to recover that data. Candidates do this to 'fix' a stuck pod.
What to do instead
Scale down through the operator (or run nodetool decommission) and wait for the node to leave the ring before the PVC is removed. If a pod is stuck, investigate events and logs rather than deleting it; keep the PVC until the decommission completes and verify nodetool status shows the expected node count and no unreplicated ranges.