STRICT mTLS Migration Versus Plaintext Batch Jobs and a JWT Policy

STRICT is a destination-side statement: the retail pricing sidecar will reject plaintext. So the migration is driven by who still speaks plaintext into retail, not by who has a sidecar.

1. Enumerate plaintext sources before changing anything. Query istio_requests_total{reporter="destination",destination_service_name="pricing",connection_security_policy="none"} and break it down by source workload. Also run istioctl x describe pod <pricing-pod> -n retail to see the effective peer authentication. If legacy-batch is the only plaintext source, you have one exception to design, not forty.
2. Apply namespace-level STRICT in retail, with a scoped exception. Create a PeerAuthentication in retail with mtls.mode: STRICT and portLevelMtls: {8080: mode: PERMISSIVE}; alternatively use a selector to exempt only the pricing workloads legacy-batch calls. Namespace-level policy applies to workloads in that namespace; mesh-wide policy must live in the root namespace (istio-system) with no selector - do not confuse the two.
3. Prove enforcement is real. After the change, new connections from sidecar-equipped callers should report connection_security_policy="mutual_tls"; only legacy-batch should still show none, and only on 8080. Confirm with istioctl x describe pod and destination-side metrics.
4. Retire the exception. Either give legacy-batch sidecars, front it with an egress proxy, or move it to HTTPS/mTLS. Then remove the port-level exception and finally set mesh-wide STRICT.

Order matters: setting STRICT first and adding exceptions afterwards creates an outage window you cannot undo for in-flight connections. Keep the exception as the last thing removed, and put a calendar check on the 90-day retirement date.

They are almost certainly different failures that happen to share an error string.

Legacy-batch, 100% failure. There are two candidates, and both are expected under this configuration:

• PeerAuthentication is still PERMISSIVE mesh-wide, so plaintext arrives without a peer identity. If the policy also matches on source.principals, plaintext requests have no principal and are denied.
• No sidecar means no JWT validation happens at the workload; request.auth.claims is never populated, so the when condition cannot be satisfied and the deny-all tail of the policy applies.

A 100% failure with a stable input is a policy-shape problem, not a flake.

Frontend, ~4% failure. A valid JWT plus a sporadic 403 suggests something varies per request or per pod: token expiry/clock skew at the edge of the token lifetime, a subset of frontend pods running an older image that sends a different audience or omits the groups claim, or some frontend replicas talking to a differently configured revision. Check the 403 rate broken down by source pod (istio_requests_total{response_code="403"} grouped by source workload) - if it is concentrated on a few pods, it is the callers, not the policy.

Separate them with istioctl x authz check <pod> -n retail, which lists every policy that applies to the workload and shows which one denies a given request shape, plus proxy logs filtered for RBAC. Fix legacy-batch by giving it an identity or an explicit IP-scoped exception; fix frontend by confirming the token contents (iss, aud, groups) actually match the RequestAuthentication and policy conditions for every replica.

An allow rule for an unauthenticated caller can only match on network attributes, because there is no cryptographic identity to match on.

Shape of the rule, in the retail namespace, ordered before the deny:

• from: source: ipBlocks: [<legacy-batch pod CIDR>] - use the actual pod CIDR or the specific addresses, not the whole cluster range.
• to: operation: methods: ["GET"], paths: ["/api/cart", "/api/pricing/*"] and ports: ["8080"], so the exception cannot be reused for other endpoints.
• Keep it as a separate, clearly named policy with a comment and an owner, so it is deletable in one object when the jobs retire.

Prerequisite: STRICT mTLS at that port would reject the plaintext connection before authorization is consulted. The allow rule only matters if port 8080 is PERMISSIVE - via portLevelMtls in the retail PeerAuthentication or by leaving the namespace policy PERMISSIVE for that port. This is why "add an allow rule" alone does not fix a 403-turned-into-TLS-failure situation; you must change both layers.

Limits to state explicitly to security:

• IP-based rules are spoofable from inside the cluster network and break if the jobs are rescheduled onto nodes with a different CIDR.
• The rule is invisible to audit: you cannot tell which human or job made the call.
• It provides no protection against a compromised workload in the allowed range.

Therefore treat it as a dated exception with the 90-day retirement as the removal trigger, and add an alert on the rule's request volume so the exception cannot quietly become permanent.