Cluster Mesh connects but global Service uses only local backends
Two Cilium 1.15 clusters, east and west, are connected with Cluster Mesh. Both were built from the same Terraform module: pod CIDR 10.244.0.0/16, service CIDR 10.96.0.0/16. Cluster Mesh status on east says west: connected, and cilium clustermesh status shows remote nodes but also a warning: 'Overlapping PodCIDRs detected'. A Service payments in namespace prod is annotated service.cilium.io/global: "true" in both clusters. East has 3 local backends; west has 2. East pods resolve the service and connect, but all traffic goes to east backends. cilium service list on east shows only the 3 local backends. The team needs cross-cluster failover but cannot renumber both clusters at once. They propose using the same service CIDR as the pod CIDR to simplify addressing.
1. Explain why Cluster Mesh reports connected but the global Service does not use west backends.
Read the worked answer
Cluster Mesh needs unique pod CIDRs across clusters. Both clusters use 10.244.0.0/16, so a west pod and an east pod can have the same IP, for example 10.244.3.17. The control plane can still connect and exchange service metadata, which is why cilium clustermesh status shows connected and remote nodes. But the dataplane cannot safely install remote endpoint routes or backends: the same prefix already belongs to local pods. Cilium therefore keeps the global Service local-only, and cilium service list shows only east backends. The 'Overlapping PodCIDRs detected' warning is the direct explanation; the connected control plane is not proof that cross-cluster dataplane is usable.
2. What remediation options does the team have, and what are the trade-offs?
Read the worked answer
Options:
- Renumber west to a non-overlapping pod CIDR such as
10.245.0.0/16. This is the clean fix. It requires rebuilding nodes or recreating pods with new addresses, plus updating static routes and any policies that reference pod IPs. - Build a new west cluster with the new CIDR and migrate workloads, then remove the old cluster. This avoids in-place renumbering but costs a second cluster during migration.
- Do not try to use Cluster Mesh with the overlap; it is not a supported workaround. Segmenting with policies does not fix the IP ambiguity. If cross-cluster failover is urgent, use a higher-level mechanism such as Gateway API or DNS-based failover between independent clusters until one side is renumbered. The proposal to reuse the service CIDR as the pod CIDR would make the overlap worse and should be rejected.
3. Which checks would confirm the overlap and verify that global service export/import works after remediation?
Read the worked answer
Confirm overlap with kubectl get ciliumnode -o custom-columns=NAME:.metadata.name,PODCIDR:.spec.ipam.podCIDRs in both clusters and compare against each cluster's pod CIDR. Run cilium-dbg clustermesh status on both sides and look for the overlap warning and remote service counts. Check cilium-dbg service list for the global Service: local-only backends confirm the dataplane did not import remote endpoints. Use cilium-dbg endpoint list to see whether remote endpoints appear. After renumbering, run cilium connectivity test --multi-cluster and verify the global Service has backends from both clusters, then test failover by scaling west to zero and confirming east still serves.