External Clients Bottlenecked Behind a Single LoadBalancer VIP
An external reporting service outside the Kubernetes cluster connects to Cassandra through a single LoadBalancer Service that selects all 6 Cassandra pods on port 9042. The Java driver 4.x is configured with that one VIP as its contact point and uses LOCAL_QUORUM. The reporting team reports that throughput is capped at about 40 MB/s even though each Cassandra node can serve 300 MB/s, and one node shows 90% CPU while the others are at 15%. During a rolling restart, the service sees NoHostAvailableException for 40–60 seconds even though the operator restarts only one pod at a time and readiness probes pass. The platform team proposes raising the load balancer's connection limit and adding a second VIP pointing at the same pods.
1. Explain the traffic distribution problem and why adding a second VIP to the same pods will not fix it.
Read the worked answer
The driver is using the VIP as a single host. It connects to the LoadBalancer, receives the cluster topology from system.peers, and learns the pod IPs. External clients cannot route to those pod IPs, so the driver either fails to connect to the peers or never uses them; the only usable connection is the one TCP connection to the VIP, which the L4 load balancer hashes to one pod. That explains one node at 90% CPU and the others at 15%.
Adding a second VIP gives the driver a second contact point, not a topology. It may open a second connection to a second randomly chosen pod, but it still cannot address all six nodes, still cannot do token-aware routing across the ring, and still has a single connection as a bottleneck. The correct fix is to give the driver a routable address per pod: per-pod LoadBalancer or NodePort services, a headless service plus external DNS and network routing to pod IPs, or an L4 proxy that exposes each pod separately and a driver AddressTranslator that maps pod IPs to those external addresses. Then the driver can pool connections to every node and route token-aware requests.
2. What causes the 40–60 second outage during a rolling restart, and how do readiness, endpoint removal, and driver reconnection interact?
Read the worked answer
The outage is the gap between the pod being deleted and the driver moving its connection. The driver's control connection or pooled connection is on the restarting pod. Kubernetes removes the pod from the Service endpoints, but the external load balancer may keep sending new connections to that backend until its health check fails. Existing TCP connections are not drained; they are blackholed when the pod's process exits. The driver waits for its heartbeat interval and socket timeout before marking the host down, then reconnects through the VIP, which may still point at a terminating endpoint. Readiness probes do not help if they pass before the node has left the ring or if the load balancer ignores readiness.
To fix it: add a preStop hook or operator drain step that stops client traffic before shutdown, use a readiness probe that checks CQL native transport, ensure endpoint removal and load-balancer health checks are fast, set sensible driver connect and heartbeat timeouts, and use a retry policy that tolerates one host down. With per-pod addressability the driver keeps connections to the other five nodes and the restart is invisible.
3. Design the target connectivity architecture for external clients with a 99.95% availability target, and state two trade-offs.
Read the worked answer
For 99.95%, do not rely on a single VIP. Option A is one external Service per pod, selected by the StatefulSet pod-name label, plus a driver AddressTranslator that replaces pod IPs with the corresponding external addresses. This gives the driver all six hosts, preserves token-aware routing, and lets it fail over during a rolling restart. The trade-off is cost and operational overhead: six load balancers, six IPs to allowlist, and certificates that include each external address.
Option B is to expose the headless service through external-dns and route pod IPs to clients over a VPN or peering. This is cheaper and preserves topology, but it requires routable pod CIDRs and a network security review. Option C is an L4 proxy tier with per-pod backends and a stable endpoint, plus driver address translation; this adds another component to scale and monitor. In all cases, keep TLS hostname verification aligned with the addresses the driver actually uses, set LOCAL_QUORUM and a retry policy, and test a full rolling restart with the external client before declaring 99.95%.