TLS Rollout Breaks Internode Joins and External Driver Topology Refresh
After a security review, you enable TLS for internode and client-to-node traffic on a 6-node cluster (3 racks, RF=3). cert-manager issues one certificate per pod from an internal CA, but the CSR template only includes DNS SANs for *.cassandra.svc.cluster.local; the pods' broadcast_address values are pod IPs. Rolling restart begins; four nodes come up, but two remain in a crash/restart loop with SSLException: No subject alternative names matching IP address 10.244.3.17 present. Client applications using the Java driver 4.x with a truststore connect to the cassandra ClusterIP Service but then fail with NoHostAvailableException after the driver refreshes its topology. A monitoring exporter that uses JMX without TLS still works. The incident commander wants to disable TLS entirely to restore service.
1. Explain the two distinct TLS failures and why they share a root cause. What is the least-risk fix that keeps encryption on?
Read the worked answer
Both failures come from certificate identity, not from encryption itself. The certs contain DNS SANs for *.cassandra.svc.cluster.local, but Cassandra internode connections use each pod's broadcast_address, which is the pod IP. With endpoint verification enabled, the joining node compares the peer certificate to the IP it connected to and finds no IP SAN, so the TLS handshake fails and the node cannot join.
The Java driver has the same problem one layer up: it connects to the Service DNS name successfully, then reads system.peers and tries to open connections to pod IPs. Those IPs are not in the certificate SANs, so hostname verification fails and the driver marks hosts down until only the contact point remains or it throws NoHostAvailableException. Keeping encryption on, the least-risk fix is to make certificate identity match the addresses actually used. For internode traffic inside the cluster, either issue certificates with IP SANs for each pod, which is hard with dynamic pod IPs, or disable endpoint verification for internode only and rely on network policy and a private pod network. For clients, issue IP SANs or use per-pod stable DNS names with a driver address translator. Disabling TLS entirely fixes nothing and creates a larger exposure.
2. The exporter still works over JMX. Is that a security gap, and how do you close it without breaking the rollout?
Read the worked answer
Yes, the working JMX exporter is a gap. JMX on 7199 is a separate channel from the native transport; enabling TLS on 9042 does nothing for it. If JMX is unauthenticated and plaintext, anyone who can reach the pod can read MBeans and, depending on configuration, invoke operations. I would not fix it in the same change as the native TLS rollout because that would mix two failure domains.
Interim control: a NetworkPolicy that allows 7199 only from the Prometheus namespace, plus disabling remote JMX if the exporter can run as a sidecar and connect over localhost. Then, as a separate change, enable JMX authentication and TLS, or switch to a sidecar that exposes only the metrics you need. Roll one rack at a time and update the scrape configuration. Keep the exporter credentials in a Secret and rotate them. Document that JMX is an administrative interface, not a public metrics endpoint.
3. Design the safe completion sequence from the current partial state, including certificate rotation and client cutover.
Read the worked answer
Freeze the rollout at the current state. Do not restart more nodes until the certificate template is corrected. Reissue certificates with the identities you chose, verify them with openssl x509 -text for the expected SANs, and test on one canary node in a rack where the other nodes are healthy. After the canary rejoins, run nodetool status, check logs for TLS errors, and perform a CQL read and write with the client truststore.
Then proceed one node at a time, never taking two nodes from the same rack, and watch the driver's host count and error rate. Update client truststores before switching the client endpoint, and keep the old truststore available for rollback. If you disabled endpoint verification for internode, record that as an accepted risk with a compensating control such as network policy plus mTLS for clients. Finally, plan certificate rotation as a recurring operation: short-lived certs are only safe if the rotation procedure has been tested.