Operations, Monitoring, Troubleshooting, and Upgrades
This domain tests day-two operations: reading nodetool and logs, diagnosing unavailable or latency issues, safely decommissioning or replacing nodes, and performing rolling upgrades.
Monitoring and diagnosis
- nodetool status shows UN for up/normal, DN for down/normal, UL for up/leaving, and UJ for up/joining. tpstats reveals dropped messages and blocked stages; compactionstats shows compaction backlog; tablestats shows SSTable counts, tombstone stats, and latency; netstats and gossipinfo help with streaming and membership.
- Check system.log and debug.log for exceptions, gossip state, and OOM or GC. JMX metrics feed Prometheus and Grafana; track heap, GC pause, disk, pending compactions, hinted handoff, and dropped mutations.
- Disk full is a common outage: commitlog and data directories fill, and snapshots and old SSTables consume space. Do not delete files manually without understanding snapshots.
Safe changes
- nodetool decommission streams ranges to other nodes and leaves the ring. nodetool drain flushes memtables and stops gossip; use it before shutdown, not as general maintenance. nodetool removenode removes a dead node token ranges without streaming from it.
- Rolling upgrade: upgrade one node at a time, verify UN and no streaming or compaction backlog, then proceed. Run nodetool upgradesstables when release notes require it. Do not bootstrap or decommission during an upgrade.
Worked example
A node is DN and will not return. First confirm it is truly dead with nodetool status, ping, and logs. If permanent, run nodetool removenode host-id from a live node, then repair affected ranges. If transient, restart it and let hinted handoff and repair catch up; do not removenode a node that will rejoin, or risk data loss and duplicate tokens.
Traps
Running removenode on a live node; using decommission for a dead node; upgrading all nodes at once; ignoring tpstats dropped messages; and deleting commitlog or SSTable files to free disk without checking snapshots.