Strict mTLS Reveals a Service Bypassing the Mesh
A company runs Consul 1.17 with service mesh across two EKS clusters and 12 EC2 virtual machines. There are 45 services, and mTLS is currently in permissive mode with a default intention of allow. One team exposes its checkout service through a Kubernetes NodePort because a legacy partner cannot use the mesh ingress gateway. A recent security review found that the partner traffic reaches checkout pods without passing through a sidecar, so it is not covered by intentions. After a sidecar rollout, checkout p99 latency rose by 40 ms and two upstream health checks began flapping. The team wants to move to strict mTLS next month but cannot break the partner integration or the VM services.
1. What is the safe sequence to move from permissive to strict mTLS without an outage?
Read the worked answer
Inventory first: identify every service instance, including the 12 VMs, that has a sidecar and a valid Consul identity. Confirm that all service-to-service traffic uses mesh ports, not direct pod IPs or NodePorts. In permissive mode, check proxy metrics and intentions for plaintext traffic. Fix the partner path by routing it through the mesh ingress gateway, or by terminating at a controlled gateway that has a Consul identity, before enforcing strict mode.
Then enable strict mTLS cluster by cluster, starting with non-production. Use a default-deny intention only after allow intentions exist for every required dependency. Monitor Envoy connection errors and consul intention check results. For the VMs, confirm the Consul agent and sidecar are healthy and certificates are rotating. If a dependency is missing, strict mode will fail closed; keep a tested rollback to permissive. Do not enable strict mTLS and default-deny in the same change window.
2. How should the team stop the NodePort bypass while preserving the partner integration?
Read the worked answer
The NodePort path is outside the mesh, so Consul intentions cannot identify or authorize it. Replace it with a mesh ingress gateway that terminates the partner connection and originates an mTLS connection to checkout with a defined gateway identity. Lock down the checkout pods with a Kubernetes NetworkPolicy that allows ingress only from the ingress gateway and mesh sidecar ports, not from arbitrary node ports. Remove the NodePort service or restrict it at the security group level.
For the partner, provide the gateway address and any required client allowlist. Add an intention allowing the ingress gateway service to call checkout, and deny other unintended sources. If the partner cannot use mTLS, terminate TLS at the gateway and enforce network restrictions there; the mesh hop from gateway to checkout should still be mTLS. Test with the partner before removing the old path and keep both paths only during a short, monitored cutover.
3. What are plausible causes of the 40 ms latency increase and flapping health checks after the sidecar rollout?
Read the worked answer
A 40 ms increase is large for a local sidecar hop, so look for an extra network path rather than normal proxy overhead. Cross-cluster traffic may be routing through a mesh gateway in the wrong region or through a NAT path; health checks may be hitting a different listener or being redirected through the transparent proxy. Check whether the sidecar is intercepting the health-check port, whether readiness probes use the pod IP instead of localhost, and whether upstream connection pooling is disabled, causing a new mTLS handshake per request.
Review Envoy statistics for upstream connect time, pending requests, and TLS handshakes. Confirm that the two flapping upstreams are registered with the correct health-check port and that their Consul health checks pass from the mesh. If the sidecar intercepts kubelet probes, exclude the probe port or adjust the probe to the proxy's exposed endpoint. Also check for CPU limits on the sidecar; throttling can add latency and cause intermittent failures.