Canary Control-Plane Upgrade Stalls at Sidecar Injection
Your production cluster runs Istio 1.20.4 installed with the default revision. The mesh has 3 istiod replicas, 42 namespaces, about 900 sidecar-injected pods, strict mTLS, and a single ingress gateway. A change freeze allows only a control-plane canary upgrade to 1.22.1. You install the new control plane with revision=canary and confirm its istiod pods are healthy. You then label the checkout namespace with istio.io/rev=canary but leave the existing istio-injection=enabled label in place. Ten minutes later, newly created checkout pods still report proxy version 1.20.4 in istioctl proxy-status. A Kubernetes Job created after the label change has no sidecar; its pod template carries sidecar.istio.io/inject: "false" inherited from a shared chart. istioctl analyze reports no validation errors. The platform team wants to finish the canary migration this window and needs a rollback path that does not require reinstalling the old control plane.
1. Explain why the new checkout pods still receive the 1.20.4 proxy and why the Job pod has no sidecar. What exact labels, annotations, and webhook behavior should you inspect before changing anything?
Read the worked answer
The namespace has two injection signals at once. istio-injection=enabled matches the default-revision injector webhook, while istio.io/rev=canary matches the canary injector webhook. With both present, the default webhook can still mutate the pod template first, so the pod is injected with the 1.20.4 proxy even though the namespace also carries the canary label. The Job is a separate case: an explicit sidecar.istio.io/inject: "false" annotation on the pod template overrides namespace-level injection. That is why it gets no sidecar despite the namespace label.
Inspect before changing:
kubectl get ns checkout --show-labelsto confirm both labels are present.kubectl get mutatingwebhookconfigurations | grep istioand read thenamespaceSelectorfor the default and canary webhooks. Both selectors must not match the same namespace if you expect a clean revision choice.kubectl get pod <job-pod> -o yamlto confirm the explicit injection-false annotation came from the chart.istioctl experimental check-inject -n checkout <pod-or-deployment>to see which revision the injector selects.
Fix the namespace by removing istio-injection=enabled and keeping only istio.io/rev=canary. Restart Deployments so new ReplicaSets are created. For the Job, remove the inherited annotation or accept that it is outside the mesh; deleting and recreating the Job is required because pod template annotations are immutable.
2. Describe the rollout sequence to move all namespaces to 1.22.1 and then make canary the default, including how to roll back a namespace that shows elevated 5xx after restart.
Read the worked answer
Treat the revision label as the unit of migration. For each namespace, remove istio-injection=enabled, add istio.io/rev=canary, and restart one workload at a time. Watch istioctl proxy-status to confirm the proxy version flips, then watch error rate and latency for that workload before moving to the next. Do not restart all 900 pods at once; the old istiod must remain running because un-migrated proxies still connect to it.
After all namespaces are on the canary revision, move the default tag only if you rely on tag-based selection: istioctl tag set default --revision canary. If namespaces select revisions explicitly, the tag change is not required, but it keeps future unlabeled injection predictable. Restart the ingress gateway pods with the canary revision label as a separate step and verify listener and route configuration before sending production traffic.
Rollback is per namespace and must not require reinstalling the old control plane. Set istio.io/rev=default on the affected namespace, restart the workload, and verify the proxy returns to 1.20.4. If the default tag was already moved, set the namespace label to the old revision name explicitly rather than relying on the tag. Keep both istiod revisions installed until no proxy reports the old version and the canary has survived a full traffic cycle. Then uninstall the old revision.