Cassandra K8s cheat sheet

A StatefulSet guarantees stable pod identity: ordinal names (cassandra-0..N-1), stable DNS (pod.service.namespace.svc), and stable PVC binding per replica.

Ordered pod management: default OrderedReady creates pod N only after N-1 is Running and Ready; podManagementPolicy: Parallel removes that serialization but loses deterministic bootstrap order.

Deployments are unsuitable for Cassandra because pods are interchangeable and share no stable identity or storage attachment.

A headless Service (clusterIP: None) is required for StatefulSet DNS; each pod gets an A record, and with publishNotReadyAddresses: true records exist before readiness.

StatefulSet scale-down deletes the highest ordinal first and does NOT delete its PVCs by default; persistentVolumeClaimRetentionPolicy controls that behavior.

The partitioner must be consistent across all nodes; a pod template change to num_tokens or partitioner requires a full cluster rebuild, not a rolling update.

Node affinity or topologySpreadConstraints should spread replicas across zones/racks; anti-affinity on hostname prevents two Cassandra pods sharing a node.

PodDisruptionBudgets limit voluntary evictions during drains; with minAvailable equal to RF, node drains block until replicas are rescheduled.

Partition key determines which node owns the data via consistent hashing (Murmur3Partitioner); clustering columns determine on-disk sort order within a partition.

Replication factor and consistency level interact: QUORUM reads/writes need RF/2+1 replicas; LOCAL_QUORUM stays within a single datacenter.

Hinted handoff, read repair, and anti-entropy repair (nodetool repair) are the three mechanisms that restore consistency after failures.

A partition should stay under roughly 100 MB and 100k rows; wide partitions cause GC pressure, read timeouts, and repair amplification.

SSTables are immutable; compaction strategies (STCS, LCS, TWCS) trade write amplification against read amplification and space amplification.

Cassandra is eventually consistent and last-write-wins per cell by timestamp; use lightweight transactions (IF NOT EXISTS) sparingly because they use Paxos.

ALLOW FILTERING and non-key secondary index queries scale poorly; design tables per query rather than modeling around entities.

Snitches (GossipingPropertyFileSnitch, Ec2Snitch) map nodes to datacenters and racks, which drives replica placement and repair scope.

The operator runs as a Deployment in its own namespace and watches CassandraDatacenter custom resources cluster-wide or namespace-scoped.

Install order: CRDs first, then the operator Deployment/RBAC, then the CassandraDatacenter resource; CRD upgrades must precede operator image upgrades.

The CassandraDatacenter spec defines serverType, serverVersion, size (nodes per rack), racks, storageConfig, and resources; changes are reconciled, not applied blindly.

Reconciliation is level-triggered: the operator compares desired spec to actual StatefulSets, Services, ConfigMaps, and PVCs and converges them.

serverVersion upgrades are gated by the operator, which checks cluster health and schema agreement before advancing to the next node.

Deleting a CassandraDatacenter does not automatically delete PVCs; reclaim policy and finalizers determine whether data survives.

Operator and Cassandra version compatibility is version-specific; check the operator release notes for the supported serverVersion matrix before upgrading.

Status conditions (Ready, Updating) and the CassandraDatacenter status field are the authoritative signal that reconciliation finished.

A rack is the failure domain for replica placement; with RF=3 use three racks (or three zones) so replicas never share a failure domain.

Racks are defined in the CassandraDatacenter spec as a map of rack name to node count; each rack becomes its own StatefulSet in the operator.

The snitch must match the topology: GossipingPropertyFileSnitch reads cassandra-rackdc.properties; a mismatch causes replicas to be placed in the wrong domain.

num_tokens (vnodes) is set per node at bootstrap and cannot be changed without rebuilding the node; 16 is a common default, 1 is used with allocation tokens.

seed nodes are contact points for gossip only; do not list all nodes as seeds, and never make every node a seed in large clusters.

Adding a datacenter uses a separate CassandraDatacenter with its own rack definitions and a rebuild from an existing DC, not a simple scale-up.

Heap and off-heap sizing: set MAX_HEAP_SIZE and heap_newgen; leave the rest of node memory for page cache and off-heap structures.

Cassandra 4+ requires Java 11; Cassandra 5 requires Java 11 or 17. The operator injects the correct image for the chosen serverVersion.

storageConfig in the CassandraDatacenter creates a volumeClaimTemplate; each pod gets its own PVC named data- (or per the template).

Use a StorageClass with volumeBindingMode: WaitForFirstConsumer so the PV is provisioned in the same zone as the scheduled pod.

Cassandra requires local or block storage; NFS and network filesystems are unsupported for data directories because of fsync semantics.

Right-size PVCs up front: most CSI drivers support expansion but not shrinking, and expansion is online only with allowVolumeExpansion: true.

The operator can add a new PVC to an existing pod via additionalVolumes, but the pod must be restarted to attach it.

Backups should use the CassandraDatacenter's storageConfig for the data volume and a separate volume for commitlog if configured.

Deleting a PVC deletes the data; the StatefulSet does not own PVCs unless persistentVolumeClaimRetentionPolicy is set to Delete.

Pod eviction during a node drain respects the PDB; if the PDB blocks eviction, the drain hangs until a replica is rescheduled.

The operator creates a headless Service per rack for peer discovery and a client Service for drivers; clients should target the client Service, not pod IPs.

The driver uses the contact points to discover the full topology; the node's broadcast_address and broadcast_rpc_address must be routable from clients.

In Kubernetes, set broadcast_rpc_address to the pod IP or use a Service per pod; NAT between clients and nodes breaks driver topology discovery.

native_transport_port defaults to 9042; internode ports are 7000 (storage) and 7001 (SSL). Expose 9042 only to client networks.

A LoadBalancer or NodePort Service exposes CQL externally; a ClusterIP Service is sufficient for in-cluster clients.

NetworkPolicies should restrict internode traffic to Cassandra pods and allow 9042 only from application namespaces.

DNS: pods resolve peers via ...svc.cluster.local; CoreDNS must be healthy or gossip bootstrap stalls.

The operator configures seeds from the headless Services; changing seed lists is handled by reconciliation, not manual edits to cassandra.yaml.

Internode TLS is configured via serverEncryptionOptions in the CassandraDatacenter spec; client TLS via clientEncryptionOptions.

Certificates can be supplied as a Secret referenced by the spec, or generated by cert-manager; the operator mounts them into the pods.

require_client_auth: true enforces mutual TLS; without it, clients connect over TLS but are not certificate-authenticated.

Authentication uses PasswordAuthenticator or MutualTLSAuthenticator; authorization uses CassandraAuthorizer or AllowAllAuthorizer.

Default superuser credentials come from a Secret (often cassandra-superuser); rotate by updating the Secret and running the operator's credential rotation.

Roles are hierarchical: CREATE ROLE, GRANT role TO role, and GRANT permission ON resource TO role; permissions are per keyspace/table.

JMX is often secured with password or SSL authentication; the operator can enable JMX auth via config.

Kubernetes RBAC on the CassandraDatacenter resource is separate from Cassandra roles; a user with kubectl edit access can change the cluster.

nodetool snapshot creates hard links to SSTables; it is fast and cheap but only valid while the SSTables are not compacted away.

A consistent backup requires a snapshot on every node plus the schema; commitlog archiving is needed for point-in-time recovery.

Medusa is a common Cassandra backup tool that integrates with object storage (S3/GCS) and supports full and incremental backups.

Restore order: stop the node, clear the data directory, copy SSTables into the table directory, run nodetool refresh, then repair.

Snapshots are not a substitute for repair; restoring from snapshot on one node requires a repair to reconcile with surviving replicas.

The operator's backup integration (for example via a CassandraBackup CR or Medusa sidecar) coordinates snapshots across racks.

Test restores into a separate namespace; a backup that has never been restored is unverified.

Disaster recovery across regions relies on a second datacenter with RF configured and rebuild, not on copying SSTables between clusters.

Metrics sources: JMX (via jmx_exporter or the operator's metrics sidecar), nodetool status/tpstats, and Cassandra's own system tables.

Key metrics: read/write latency p99, pending compactions, dropped mutations, hinted handoff, live/total disk, and GC pause time.

nodetool tpstats shows dropped messages per stage; nonzero dropped mutations indicate the node is saturated.

Metric endpoints are scraped by Prometheus; the operator can create ServiceMonitors or annotations for discovery.

Logs go to stdout/stderr and are collected by the cluster logging stack; set logback levels via the CassandraDatacenter config.

Alerts should cover: node down, high pending compactions, disk above 80%, and repair overdue (gc_grace_seconds exceeded).

nodetool tablestats reports SSTable counts and read/write latency per table; use it to find wide partitions and hot tables.

The operator exposes reconciliation status and errors as Kubernetes events and status conditions; watch those, not just Cassandra metrics.

Scale up by increasing rack size in the CassandraDatacenter; the operator adds one node at a time and waits for it to join before continuing.

Scale down removes the highest-ordinal pod; run nodetool decommission first if you need to drain data before the pod is deleted.

Rolling restart: the operator restarts pods one at a time, waiting for the node to be Up/Normal and for schema agreement before proceeding.

Upgrades change serverVersion; the operator performs a rolling upgrade and blocks if the cluster is not healthy.

Do not skip major versions: upgrade Cassandra one major version at a time and run nodetool upgradesstables after each.

Schema changes should be applied once and allowed to propagate; concurrent DDL during a rolling restart causes schema disagreement.

Before any rolling operation, run nodetool repair so the cluster can tolerate a node being down.

Watch the CassandraDatacenter status and events during operations; a stuck reconciliation usually means a pod is not becoming Ready.