One EnvoyFilter, 1,200 Proxies, and a Melted Control Plane

Raising CPU treats a symptom. The driver is a config-push feedback loop.

Chain, link by link:

1. Blast radius. No workloadSelector means the filter is attached to every proxy in the mesh, so every xDS response is larger and every proxy must recompile listeners.
2. Rejection. If any part of the 200-line patch is invalid for the installed Envoy version - a renamed filter, a missing typed_config, a bad patch context - proxies NACK it. istiod records the NACK and keeps trying, because the desired state still differs from what the proxy acknowledged.
3. Loop. Each retry recomputes and re-serializes config for the affected proxies, which is CPU- and allocation-heavy; heap growth follows. That is the 0.4 → 3.2 cores and 2.1 → 6.4Gi.
4. Contention. Push p99 rises to 12s because istiod's push queue is saturated; proxies whose pushes time out stay STALE, and the 480 number is the queue backlog made visible.

Confirm explicitly:

• kubectl logs deploy/istiod -n istio-system | grep -i -E "nack|rejected" - you want the proxy ID and resource name per failure.
• istioctl proxy-status <pod> on both a STALE and a SYNCED proxy; if only a subset is STALE, look for what those proxies share (node, version, workload type).
• istioctl proxy-config listener <pod> -o json to see that the patched filter chain is absent on affected proxies.
• Metrics: pilot_proxy_convergence_time (p99 climbing), pilot_xds_pushes by type, and process CPU/heap on istiod.

The decisive test is cheap: delete or workloadSelector-scope the EnvoyFilter and watch CPU, memory and push time fall within a couple of push cycles. If they do not, the filter is not the whole story and you look at endpoint churn next.

Work from the response flags and the reporter, because a 503 has several very different meanings.

• Reporter dimension. istio_requests_total{response_code="503",reporter="source"} vs reporter="destination". Source-reported 503s mean Envoy could not get a response from the upstream (network, no endpoints, rejected config). Destination-reported 503s mean a proxy inside the destination pod generated the response - often the app's own 503 passed through, or an inbound listener problem.
• Response flags. In istio_requests_total the response_flags label and in access logs:
  • UF - upstream connect failure: listener/cluster present but connection failed. Could be app down or a half-applied config.
  • NR - no route configured: strongly control-plane/config, e.g. a route that disappeared during a push.
  • UO - overflow: connection pool exhaustion, often the app being slow rather than the mesh.
  • - (none) with a destination reporter - normal application-generated 503.
• Correlation with push events. Plot 503 rate against pilot_proxy_convergence_time p99 and pilot_xds_pushes. If the bursts align with push spikes and the flags are NR/UF, the control plane is implicated. If 503s are flat while pushes spike, keep looking.
• Per-workload split. Group by destination_workload and source_workload. Control-plane causes tend to be broad and version-correlated (all proxies on one revision, or all workloads touched by the filter); application causes concentrate on one Deployment.
• Proxy state at the time. A STALE proxy serving 503s while its peers are SYNCED is essentially proof that the config, not the app, is the differentiator.

Finally, reproduce: apply the EnvoyFilter in a staging cluster with the same Istio version and confirm the same flags appear. A root cause you cannot reproduce is a hypothesis.

It is the same incident surfacing on a different path. Injection is an admission-time call from the Kubernetes API server to istiod's injector webhook; if istiod is saturated, that call times out.

What decides the outcome is the webhook's failurePolicy:

• Ignore - the API server continues without injection. Pods start cleanly with no istio-proxy, exactly as described. No error is shown to the user; kubectl describe pod may show a webhook warning event.
• Fail - pod creation is blocked entirely. You would see pending pods and admission errors instead.

So the missing sidecar is not a namespace label problem in this case, but verify in this order so you do not chase the wrong thing:

1. kubectl get ns reporting --show-labels - confirm istio-injection=enabled and that no istio.io/rev label overrides it (a namespace with istio.io/rev ignores istio-injection).
2. kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml - check failurePolicy, namespaceSelector, and whether the webhook's caBundle matches the current istiod (a CA rotation during a restart can invalidate it).
3. kubectl describe pod <pod> -n reporting - look for admission webhook timeout events.
4. istiod logs for inject and webhook timeouts during the window.

Remediation: fix the push loop first, then delete and recreate the affected pods so they get sidecars - a running pod never gains one. For the future, treat failurePolicy: Fail as the production default (fail closed rather than silently running outside the mesh), and alert on injection skip/failure metrics and on pods in mesh namespaces without a proxy.