I85PREP
⌘ K
Istio Certified Associate study materials
Istio Certified Associate study resources

Revision-Based Canary Upgrade Breaks One Namespace

Istio Certified Associate · Put your knowledge to work in context.

All case studies

Revision-Based Canary Upgrade Breaks One Namespace

Your platform team operates Istio 1.17.3 installed with istioctl using the default profile: three istiod replicas, an istio-ingressgateway with an HPA (min 3, max 12), 240 workloads across 12 namespaces, all injected through the namespace label istio-injection=enabled. A single shared EnvoyFilter lives in istio-system and adds a custom header to every inbound listener at the HTTP_FILTER stage. You must reach 1.20.1 with zero failed requests, and security wants 1.17 fully removed within seven days. You choose a revision-based canary install, run istioctl install --revision=1-20-1 --set revision=1-20-1, and relabel only the payments namespace with istio.io/rev=1-20-1. New payments pods start with 1.20 sidecars. Within minutes, 6% of calls into payments return 503 with response flag UF, while payments pods still running the 1.17 sidecar serve fine. istioctl proxy-status shows the new payments proxies as STALE.

1. Was `istioctl install --revision=1-20-1` enough to stand up a usable canary control plane? What must be verified or done before relabeling any namespace, and what does relabeling one namespace actually change?

Read the worked answer

The install creates a second control plane, not a replacement. Concretely it produces istiod-1-20-1, a new istio-sidecar-injector-1-20-1 webhook, and revision-suffixed gateway Deployments; the original istiod and istio-ingressgateway are untouched. Before relabeling, verify:

  • Both revisions are healthy: kubectl get pods -n istio-system and istioctl proxy-status should list both istiods, with every proxy SYNCED.
  • CRDs are shared: they are cluster-scoped, and the new install upgraded them for both control planes. Confirm the old revision still reconciles cleanly (watch istiod logs for schema errors) before sending it new work.
  • Trust domain / CA: both revisions must mount the same cacerts so 1.17 and 1.20 proxies can do mTLS with each other. Cross-revision mTLS failures look like 503s, so check this early.
  • Ingress path: traffic still enters through the 1.17 gateway. Decide deliberately whether to cut the ingress gateway to the new revision early (good: small blast radius validates 1.20 end to end) or keep it on 1.17 until every namespace migrates.
  • Default tag: istioctl tag list still points default at 1.17, so istio-injection=enabled namespaces keep getting 1.17 sidecars. Only istio.io/rev=1-20-1 selects the new revision.

Relabeling payments changes only which injector webhook handles new pods there. Existing pods keep their old sidecar until restarted, so you get a mixed data plane in one namespace - which is precisely why a shared cluster-scoped config object can break only part of it.

2. Diagnose the 503 UF and the STALE status. What is the most likely mechanism given what is shared between the two revisions, what commands confirm it, and how do you restore payments immediately?

Read the worked answer

STALE means the proxy is not acknowledging the newest config version; a 503 with UF on inbound traffic means the listener/route it acknowledged cannot serve the request. The one thing that changed for payments is the Envoy version behind a cluster-wide EnvoyFilter with no workloadSelector.

Likely mechanism: the filter patches HTTP_FILTER on every inbound listener using a schema or filter name that 1.17's Envoy accepted but 1.20's Envoy rejects. The new listeners are rejected (NACK), the proxy keeps the previous good snapshot for some listeners, and inbound requests to payments hit an incomplete listener chain → 503 UF. Because the filter is cluster-scoped, only the new-version proxies are affected.

Confirm in this order:

  1. istioctl proxy-status <payments-pod>.payments - read the version detail, not just the word STALE.
  2. kubectl logs <pod> -c istio-proxy | grep -i -E "nack|reject|config" - Envoy logs the rejected resource name.
  3. kubectl logs deploy/istiod-1-20-1 -n istio-system | grep -i nack - istiod logs the NACK with the proxy ID.
  4. istioctl proxy-config listener <pod> -n payments - the expected filter chain is missing.

Immediate mitigation: move payments back off the new revision. Set istio.io/rev to the old revision (or remove it and restore istio-injection=enabled) and restart the Deployment; injection is per-pod, so a rollback is a rollout. Then fix the EnvoyFilter: correct the typed config for 1.20 and add a workloadSelector so a bad patch can never take the whole mesh again.

3. Walk through the final seven-day removal of 1.17. What evidence do you require before running the uninstall, and what must not be deleted?

Read the worked answer

Removal is a sequence of proofs, not a single command.

  1. No data plane on the old revision. istioctl proxy-status must contain zero entries whose istiod column is the old revision. Because STALE/leftover entries happen, cross-check with kubectl get pods -A -o json and grep the istio-proxy image tag for 1.17.3. Anything found gets restarted with the new revision label first.
  2. Ingress and egress cut over. The 1.17 gateway Deployment must be replaced by the 1.20 one, and any Gateway resources must still resolve. Send real traffic through the new gateway and watch 5xx before deleting the old one.
  3. Tag and label cleanup. istioctl tag set default --revision 1-20-1 --overwrite so istio-injection=enabled namespaces land on 1.20, then remove stale istio.io/rev=1-17-3 labels. Namespaces carrying istio.io/rev ignore istio-injection, which is a common cause of "injection silently stopped working" after a migration.
  4. Uninstall the old revision only. istioctl uninstall --revision 1-17-3 removes that revision's Deployments, Services and webhook. Verify with kubectl get deploy,mutatingwebhookconfiguration -n istio-system that only 1.20 objects remain.

Do not delete: the CRDs (they are shared and still in use), the istio-system namespace, and the cacerts secret if you plan future revisions. Do not delete the old revision while its proxies are still connected - they lose xDS and behave unpredictably rather than failing cleanly.