Canary Weight Sends 10% of Checkout Traffic to an Empty Subset

UF is an upstream connection failure: Envoy selected a cluster and could not establish a connection or found no usable endpoint. With a subset route, Envoy does not fall back to the base service; it uses the cluster named for that subset, e.g. outbound|8080|v2|checkout.shop.svc.cluster.local. If the DestinationRule subset selects version: v2 but the pods carry version: v2.0, no pod ever joins that subset, the cluster has zero endpoints, and every request routed there fails instantly. Since the weight is 10%, the failure rate matches the weight almost exactly - a strong signal that the failure is per-route, not per-pod.

Prove it:

• istioctl proxy-config endpoint <client-pod> -n shop --cluster "outbound|8080|v2|checkout.shop.svc.cluster.local" - you expect ENDPOINT STATUS: no healthy endpoints (or an empty list) while the v1 cluster shows 8 addresses.
• istioctl proxy-config cluster <client-pod> -n shop | grep v2 to see the cluster exists but is empty.
• Compare kubectl get pods -n shop --show-labels against the DestinationRule selectors; the mismatch is usually a rename of the version label at packaging time.

Mitigations, fastest first:

1. Set the v2 weight to 0 (or remove the v2 route) - traffic instantly returns to 100% v1 with no pod restarts.
2. Correct the DestinationRule subset selector to version: v2.0 and re-apply; endpoints appear without touching pods.
3. Only if the label itself is wrong, relabel the Deployment and roll it; that restarts pods and is slower and riskier.

Validate the fix by re-running the endpoint query and raising the weight in small steps (1% → 10% → 50%), watching 5xx per destination workload rather than per service.

The policy applies attempts: 3 with perTryTimeout: 800ms to every route in the VirtualService, including POST /checkout whose p99 is 1.2s. That means a meaningful fraction of legitimate requests exceed 800ms per attempt. Envoy aborts the attempt, retries up to two more times, and the server may complete the original request anyway - the client sees a response, the order is charged two or three times. Retries are not idempotency-aware: Istio will happily retry a POST.

Reasoning about fixes:

• Retries must be per-route, not global. Add a route match for POST /checkout with retries: attempts: 0(or omit retries on that route) and a route-level timeout no lower than the observed tail latency. Keep retries on idempotent GETs, where they are useful.
• Per-try timeout must be above the failure you intend to retry. 800ms retries slow-but-successful work; it does not protect against real outages. If the intent is to mask transient connection failures, retry on connect-failure and reset only via retryOn, and set perTryTimeout above p99.
• Server-side idempotency is the durable control. An idempotency key on the order endpoint makes retries safe regardless of mesh configuration.
• Consider outlier detection instead of aggressive retries for failing v2 pods: ejection removes bad endpoints rather than multiplying load.

Verify with istioctl proxy-config route <pod> -n shop -o json and confirm the POST route carries the intended retry and timeout values, not the inherited top-level ones. Then run a load test above p99 and compare order IDs for duplicates before enabling the canary again.

Treat this as a mismatch between what the application dials and what the mesh knows about the host. With the ServiceEntry declaring only port 80/HTTP, the application's connection to 443 has no matching listener/route in the sidecar. Depending on outboundTrafficPolicy, an unknown port is either passed through as opaque TCP or blackholed; in both cases the client sits in connect or read until its own deadline, which is exactly the 30-second stall before the 503.

Check, in order:

1. The exact destination the app uses: https://tax.example.com:443 or an IP? istioctl proxy-config listener <pod> -n shop should show a listener for the port; absence is the tell.
2. istioctl proxy-config cluster <pod> -n shop | grep tax - confirm which clusters exist and for which port.
3. meshConfig.outboundTrafficPolicy.mode: REGISTRY_ONLY produces an immediate 503 NR, not a stall, so a stall points at a port/protocol mismatch rather than registry scoping.

Corrected shape: declare the port the app actually uses, with the right protocol, and keep resolution consistent with how the host resolves.

• port: number: 443, name: https, protocol: TLS with resolution: DNS, location: MESH_EXTERNAL, and hosts: [tax.example.com].
• Add a DestinationRule if you need SNI, connection pool limits or a TLS origination policy; for plain passthrough TLS, trafficPolicy.tls.mode: SIMPLE with sni: tax.example.com is only needed if you terminate or originate.
• Set an explicit timeout on the calling route so a future misconfiguration fails in 2s rather than 30s.

Re-verify by watching istio_requests_total{destination_service="tax.example.com"} and the sidecar's cluster status after applying.