LoadBalancer Service Stuck Pending on an NKP AHV Cluster
An NKP workload cluster on AHV runs 12 worker nodes and hosts an e-commerce API exposed as a Service of type LoadBalancer. Cilium is the CNI. The cluster was installed with kube-vip providing only the control-plane virtual IP. After deploying the Service, EXTERNAL-IP stays pending for 25 minutes. A NodePort test reaches the API from inside the data center, so the pods and service data path work. The platform team has a routable /24 VLAN but no BGP peering from the cluster. Security requires that users never connect through NodePort and that the virtual IP survives a worker-node failure. Traffic is HTTP and HTTPS at roughly 3,000 requests per minute, with TLS terminated at an ingress controller.
1. Diagnose why the Service has no external IP and list the checks that separate CNI behavior, load-balancer controller behavior, and address-pool configuration.
Read the worked answer
A LoadBalancer Service needs a controller that allocates an external address and programs the data path. The NodePort success proves that kube-proxy or Cilium service translation and pod connectivity are working; the missing piece is the load-balancer implementation.
Check the Service events first for messages such as no available IPs or no controller. Inspect the Cilium configuration for LB IPAM or service load-balancer capabilities; if Cilium was installed without an address pool, it will not assign an external IP. Check whether kube-vip is configured only for the control-plane VIP and not for Services. Look for a cloud-controller-manager or MetalLB controller pod and its logs. Confirm that any configured pool is in the routable VLAN, excludes the node and VIP addresses, and has free addresses. Also confirm the Service has the intended loadBalancerClass; a mismatch means no controller claims it.
The absence of BGP is not the fault here. It only rules out BGP-based designs. The team should decide on an L2-based address announcement or an external load balancer, then configure it deliberately.
2. Compare three viable designs: MetalLB in L2 mode, kube-vip service mode, and an external hardware load balancer using NodePort. Recommend one for the stated constraints.
Read the worked answer
MetalLB L2 mode assigns addresses from a pool and elects one node to answer ARP for each Service IP. It needs no BGP, integrates with Kubernetes Service objects, and fails over when the leader node is lost, usually within seconds to tens of seconds. It is simple to operate but the elected node carries the ingress traffic, so size that node and test failover.
kube-vip service mode is similar in that it can advertise Service VIPs with ARP and can coexist with the control-plane VIP. It reduces the number of components if kube-vip is already deployed, but it must be explicitly configured for Services and its leader election and failover behavior must be tested. Mixing control-plane and service VIP responsibilities needs clear ownership.
An external hardware load balancer with NodePort keeps traffic off the cluster control path and offers mature health checks, but it exposes NodePorts on every node and adds a device dependency. Security's no-NodePort-for-users rule can still be met if users hit the external VIP and the NodePorts are restricted to the load balancer, but the rule's intent may be violated.
For a /24 without BGP and a requirement for a stable VIP, MetalLB L2 in front of the ingress controller is the most direct fit. Place the ingress controller behind the Service, use the pool from the routable VLAN, and restrict NodePort access.
3. What failover and validation tests should be run before production cutover, and what result would make the design unacceptable?
Read the worked answer
Test the failure modes, not just the happy path. Resolve the Service VIP from a client outside the cluster, confirm it maps to the elected node, and verify TLS and HTTP through the ingress. Then power off or cordon and drain the node currently advertising the VIP. Measure how long until ARP converges and the VIP answers again, and confirm in-flight requests fail over cleanly. Repeat by rebooting a node and by scaling the ingress controller. Check that no single worker is a permanent bottleneck and that the pool has enough addresses for blue-green ingress.
The design is unacceptable if the VIP does not recover without manual intervention, if recovery takes longer than the agreed outage budget, if the VIP lands on a node that cannot reach the ingress pods, or if the only working path requires users to hit NodePorts. Also reject it if the address pool overlaps the node network, the control-plane VIP, or DHCP ranges. Validate that Cilium network policies still apply to traffic arriving through the load balancer and that source IP preservation is understood. Document the elected-node behavior, the pool, and the failover runbook before cutover.