Operations, Maintenance, and Troubleshooting
This domain tests day-2 operations: topology changes, repair, snapshots, monitoring, and troubleshooting exceptions. Expect command identification and reasoning about safe sequencing, not memorizing every nodetool flag.
Health and topology
nodetool status shows ownership, load, and up/down state. nodetool info reports gossip, heap, and caches. nodetool ring and describecluster show token ownership. nodetool tpstats exposes dropped mutations and request pools; compactionstats shows compaction backlog; netstats shows streaming. Use metrics and logs alongside nodetool, because a node can be up but overloaded.
Repair, cleanup, and snapshots
Repair reconciles replicas and should run within gc_grace_seconds. nodetool cleanup removes data no longer owned after topology changes such as adding nodes. It is not a repair. Snapshots are point-in-time copies used before upgrades or risky changes; nodetool snapshot creates them, and clearsnapshot removes them. Bootstrap adds a node and streams ranges; decommission streams ranges away and leaves the ring; removenode handles a dead node that cannot decommission.
Worked example
You add a node to a three-node DC with RF=3. After bootstrap, the new node owns ranges, but the old nodes still hold replica data they no longer own. Run nodetool cleanup on each existing node to reclaim disk. Do not run cleanup on the new node for ranges it just received. Then verify nodetool status shows normal ownership and run repair on a schedule. If a node is permanently dead and cannot be replaced quickly, nodetool removenode <host-id> reassigns its ranges; do not simply delete its data directory.
Troubleshooting patterns
Unavailable exceptions mean too few replicas met the consistency level. Write timeouts can come from slow disks, compaction backlog, or overloaded queues. Read timeouts often involve tombstones, large partitions, or paging through wide rows. Dropped mutations in tpstats indicate the node cannot keep up. Check disk space, heap pressure, GC pauses, and streaming before changing consistency.
Traps
Do not run cleanup before the topology change is complete. Do not treat repair as optional after restoring from snapshot. Do not decommission a node with insufficient free space elsewhere. Do not use removenode on a live node. Do not change gc_grace_seconds casually; it interacts with repair and deletion safety.