Combined Upgrade, Heap Change, and Scale-Out in a 90-Minute Window

The immediate blocker is the pod in r3 stuck Terminating because its node is NotReady; the operator will not move to the next pod until that pod is gone. The hand-written PDB with minAvailable: 6 is a second blocker: with three pods already unavailable, allowed disruptions is 0, so any drain of the NotReady node is rejected. Force-deleting the pod is safe only if the old container is definitely not running and the volume is detached. On a NotReady node the kubelet may be unreachable, so the pod can be stuck while the Cassandra process is still alive or while the EBS volume is still attached.

I would check the node condition, the pod's deletionTimestamp, the VolumeAttachment, and kubelet logs. If the node is unreachable and fenced, delete the pod with --force --grace-period=0, then confirm the volume detaches before the replacement starts. If the node may still be running Cassandra, forcing deletion risks two processes writing the same data; recover or fence the node first. Separately, the RF=1 analytics keyspace means one restarting node is a single point of failure for that workload.

Do one change class at a time and verify between steps. First fix the PDB: delete the hand-written PDB or align it with the operator's rack-aware maxPodUnavailable. A PDB that blocks drain during a node failure is not protecting availability. Then complete the 4.1 upgrade with the heap unchanged. Before starting, check that every keyspace has RF less than or equal to the number of racks, that no node is already down, and that backups are current. After the upgrade, verify nodetool status, nodetool version on all nodes, and application error rates.

Second, change the heap to 12 GiB and raise the container memory request and limit accordingly so the JVM, page cache, and off-heap overhead fit. The operator will perform another rolling restart. Verify GC and latency before proceeding. Third, scale out one node at a time to 12, waiting for each to join and for nodetool status to show UN. After all three new nodes are up, run nodetool cleanup on the existing nodes to remove ranges they no longer own. Do not combine the heap change with scale-out: if latency regresses, you need to know which change caused it.

The analytics keyspace is the reason those queries are failing: with SimpleStrategy RF=1, each partition lives on exactly one node, and that node is being restarted. QUORUM on the main keyspace is irrelevant to it. I would pause the rollout only if the analytics SLA requires it; otherwise continue the main upgrade and tell the analytics team that their keyspace cannot be made highly available without a schema change.

Fixing it properly means creating a new keyspace with NetworkTopologyStrategy RF=3, one replica per rack, migrating the data with sstableloader or a controlled COPY, and then cutting the analytics jobs over. ALTER KEYSPACE alone is not enough: changing RF does not move existing data until a repair streams the new replicas, and a full repair on a large table is its own maintenance event. Add a policy check that rejects SimpleStrategy in this multi-rack cluster. For this window, either accept analytics downtime or defer that workload; do not weaken the main keyspace's RF to compensate.