Cassandra Admin common mistakes
Cassandra Admin · Understand the traps and how to avoid them.
Designing tables around entities (users, orders, products) first, then rescuing the queries the application needs with ALLOW FILTERING, secondary indexes on high-cardinality columns, or joins in application code.
Relational habits carry over: candidates normalise the schema first and treat CQL as a subset of SQL, not registering that a non-key predicate forces a scan across every node in the ring.
What to do instead
Start from the query list and work backwards: one table per query, partition key built from the equality predicates, clustering columns built from the range/ordering predicates. When a second query needs a different key, write the row into a second table instead of indexing at read time. Reserve ALLOW FILTERING for ad-hoc admin queries on small tables.
Choosing a partition key by checking only that values look unique (status, country code, a raw timestamp, a device id with a handful of large tenants) and never estimating bytes per partition or writes per partition per second.
Cardinality and growth are invisible on a dev dataset of a few thousand rows; the key looks fine until one token range holds all the traffic or a single partition passes the 100 MB / 100k row guidance.
What to do instead
Before committing, compute row size x rows per partition and writes/sec per partition. Target well under 100 MB and 100k rows per partition, and add a bucketing component (time bucket, hash suffix, tenant shard) whenever the natural key is unbounded or skewed. Verify with nodetool tablehistograms after loading realistic data.
Answering durability questions as if a write is only safe once the memtable has been flushed to an SSTable, and treating hinted handoff as the thing that keeps a temporarily down replica consistent.
The memtable-flush picture is the memorable part of the write path, and hints look like replication; the commitlog's role and the hint expiry window get lost.
What to do instead
A write is durable once it is appended to the commitlog (fsynced according to commitlog_sync), long before any flush. Hints are best-effort: they expire after max_hint_window_in_ms (default 3 hours) and are not a repair substitute. A replica down longer than that window must be reconciled with nodetool repair before it is trusted again.
Computing quorum as a cluster-wide property: saying 'QUORUM survives two node failures' or 'QUORUM means two copies' without knowing the keyspace RF or which nodes hold the replicas.
RF, consistency level and node count get conflated, and in a dev cluster where RF equals the node count the arithmetic never actually bites.
What to do instead
Compute per keyspace and per partition: QUORUM = floor(RF/2)+1 replicas of that partition, and write CL + read CL > RF for strong consistency. With RF=3, QUORUM=2 tolerates one replica unavailable; use LOCAL_QUORUM per datacenter and remember CL=ONE or ANY gives no read-your-write guarantee.
Provisioning by listing every node in seed_provider, leaving cluster_name at its default or inconsistent between nodes, and leaving num_tokens or allocate_tokens_for_local_replication_factor at whatever the sample file shipped with.
The sample cassandra.yaml starts a single test node perfectly well, so the seed list and token allocation get treated as boilerplate rather than as topology decisions.
What to do instead
Seeds are a small, stable gossip entry point (two or three nodes that are not all the same rack), never the whole cluster; every node must carry the identical seed list and the same cluster_name. Set allocate_tokens_for_local_replication_factor to the expected keyspace RF so vnodes align with ownership, and check with nodetool status and nodetool ring after each node joins.
Switching every table to LeveledCompactionStrategy 'because it's the best one', or turning on the row cache to fix a read latency problem.
LCS is described as giving lower read amplification and the row cache sounds like a free win, so write amplification and off-heap memory cost are never budgeted.
What to do instead
Match the strategy to the workload: STCS for write-heavy and mixed, LCS for read-latency-sensitive tables with overwrite-heavy small partitions, TWCS for time-series with TTL. Keep the row cache off and tune the key cache, file cache size and read-ahead instead. Measure pending compactions and write amplification after any change.
Running nodetool repair -pr on a single node and calling it a cluster repair, or letting the repair interval drift past gc_grace_seconds because repairs are slow.
-pr only repairs the ranges that node owns, and repair feels like a background chore that can be postponed; the tie to tombstone garbage collection is not intuitive.
What to do instead
Repair every node within gc_grace_seconds (default 10 days; lower it for short-TTL tables). Schedule -pr per node in turn, or use --dc / a repair orchestrator, one keyspace or table at a time to bound impact. Never let the interval exceed gc_grace_seconds or deleted data can be resurrected.
Assuming that enabling PasswordAuthenticator and CassandraAuthorizer secures the cluster, while leaving system_auth at RF=1 and the default cassandra/cassandra superuser in place.
Authentication is the visible step; internode TLS, client TLS, audit logging and the replication of system_auth are separate settings that a lab setup never forces you to confront.
What to do instead
Change the default superuser immediately, raise system_auth RF to 3 and run repair on it, then treat authentication, authorization, server_encryption_options, client_encryption_options and audit logging as four independent tasks. Verify from a fresh client that unauthenticated and unauthorised connections are actually refused.
Reacting to one alarming metric - a pending compaction count, one slow node, or a dropped-message counter in tpstats - by changing configuration, without checking the correlated signals first.
Dashboards surface a single number and the fix feels obvious; dropped messages and pending compactions look like root causes when they are usually symptoms of tombstones, wide partitions, GC pauses or a disk throughput ceiling.
What to do instead
Triage in a fixed order: nodetool status and tpstats (dropped MUTATION/READ), pending hints, tombstone and live-cell counts from nodetool tablestats, partition size distribution from nodetool tablehistograms, then JVM GC logs and disk iostat. Change one thing at a time and confirm the metric moved before changing anything else.
Treating nodetool snapshot as a complete backup - taking it without flushing first, skipping system_auth and the schema, and later restoring the SSTables onto a cluster whose nodes no longer own those token ranges.
Snapshots look like a database dump, so it is easy to miss that they are hard links to on-disk SSTables, exclude anything still in the memtable, and carry no ownership information.
What to do instead
Snapshot every keyspace including system_auth and the schema, flush (or drain) first, and pair snapshots with commitlog archiving if point-in-time recovery is required. To restore, truncate the table, place SSTables only on nodes that own the ranges (or stream them with sstableloader when the topology differs), run nodetool refresh, then repair.