ICA: Canary Release Returns 503 UF for One Subset

Canary Release Returns 503 UF for One Subset

Your team runs a reviews service with three Deployments: v1 with 8 replicas, v2 with 4 replicas, and v3 with 1 replica. A VirtualService sends 80% to v1, 15% to v2, and 5% to v3 using DestinationRule subsets. All pods have sidecars, readiness probes pass, and the Service exposes port 8080 named http. After v3 is added, exactly the v3 share of requests fails with HTTP 503 and response flag UF; v1 and v2 remain healthy. v3 logs show no incoming requests. The v3 Deployment has labels app=reviews and version=v3, but its pod template only has app=reviews. The DestinationRule subset v3 selects version=v3. The team also enabled outlier detection with consecutive5xxErrors: 1, interval: 10s, baseEjectionTime: 30s. They want to continue the canary rather than roll back, and they need a way to prove the failure is routing-related before the next traffic shift.

1. Given the symptoms, explain why 5% of requests return 503 UF while v3 pods are Ready. What commands and Envoy configuration would you use to confirm the root cause?

Read the worked answer

The subset has no endpoints. The VirtualService weight sends 5% of traffic to the cluster for subset v3, but the DestinationRule subset selects version=v3 while the v3 pods do not carry that label. Envoy therefore sees an upstream cluster with zero healthy endpoints. Requests routed there fail before reaching the application, which is why v3 logs show nothing and v1/v2 remain healthy. UF means upstream connection failure and is consistent with no usable endpoint.

Confirm with:

kubectl get pods -n <ns> --show-labels to compare pod labels with the subset selector.
istioctl proxy-config endpoints <client-pod> -n <ns> --cluster "outbound|8080|v3|reviews.<ns>.svc.cluster.local" and check for an empty endpoint list.
istioctl proxy-config clusters <client-pod> -n <ns> to verify the cluster exists but has no healthy hosts.
istioctl proxy-config route <client-pod> -n <ns> to confirm the weighted route points at the v3 cluster.

Fix the v3 pod template to include version=v3, roll out the Deployment, then re-check endpoints before shifting any more traffic. Readiness passing is irrelevant when the endpoint never lands in the subset.

2. The outlier detection configuration is also risky for a one-replica subset. Explain the interaction and propose a safer configuration for the canary, including how you would validate it with real traffic.

Read the worked answer

With one v3 replica, consecutive5xxErrors: 1 can eject the only endpoint for 30 seconds after a single failed request. The VirtualService still sends 5% of traffic to the v3 cluster during that window, so every request routed there returns 503 UF. The canary then looks catastrophically broken even though the application may be healthy. Ejection also interacts badly with retries and short evaluation intervals because a transient failure can remove the only capacity for the subset.

Safer options:

Do not enable aggressive outlier detection on a one-replica canary. Use at least two v3 replicas and set consecutive5xxErrors: 5, interval: 30s, baseEjectionTime: 30s, and a maxEjectionPercent that leaves capacity.
Alternatively, keep outlier detection for v1/v2 only and route the v3 subset without ejection until it has enough replicas.
Fix the missing version=v3 label first; otherwise outlier detection is masking a routing defect.

Validate with a real client rather than only curl from a laptop. Run a loop from an in-mesh test pod, then query Prometheus for istio_requests_total grouped by destination_version and response_code. Check envoy_cluster_upstream_rq_retry and endpoint health in istioctl proxy-config endpoints. Only increase the v3 weight after the subset has endpoints and the error rate stays at baseline through a full observation window.