Cassandra K8s: Monitoring, metrics, and observability

This domain tests whether you can tell a healthy Cassandra node from a healthy Pod and diagnose latency with the right signals.

Core rules

• JMX exposes Cassandra metrics; a JMX exporter or agent converts them to Prometheus format.
• Useful signals include read/write latency percentiles, timeouts, unavailable exceptions, pending compactions, hinted handoff, dropped mutations, disk use, GC, and thread pool queues.
• Kubernetes signals such as restarts, OOM kills, PVC usage, and readiness complement Cassandra metrics.
• Logs and events explain why a node left the ring; nodetool status, tpstats, and tablestats provide point-in-time checks.
• Alerts should target user-visible symptoms and capacity thresholds, not only CPU.

Worked example

A p99 write latency alert fires. Pod CPU is normal, so the responder checks pending compactions and disk I/O. Pending compactions are climbing and the data volume is above 80 percent. The cause is compaction falling behind, not a client surge. The fix is capacity or compaction tuning, plus relieving disk pressure; scaling the client tier would not help. A second alert on hinted handoff shows a node was unreachable, which explains the backlog.

Traps

• Monitoring only Pod readiness.
• Exposing unauthenticated JMX broadly.
• Creating high-cardinality labels for every partition or table.
• Ignoring disk headroom and dropped mutations until writes fail.