Intermittent load balancer health check failures, CDN data leakage, and a peering request with overlapping CIDRs

Why the pattern matters. Intermittent, per-target failures with recovery rule out the obvious network faults: a wrong route, a misconfigured subnet, or a peering/ACG blackhole fails permanently and usually for everyone. Something is marginal on individual servers, or the health check is testing something that is not really liveness.

Order of investigation.

1. Reproduce the health check by hand. From inside the VPC, request the exact health check path, port, and Host header. Look at the status code, not just connectivity. A health check configured to expect 200 will fail a server returning 301 to HTTPS or 401 - and if some servers were reconfigured for HTTPS redirect, only those fail, matching the symptom.
2. Read the web server access logs for the health check requests. You will see whether requests arrive, what they return, and how long they take. If latency sits near the check's timeout, the servers are fine but too slow under load, so every check is a coin flip.
3. Check the ACG for the health check source. The check originates from the load balancer, not from the internet. Allowing 0.0.0.0/0 on 80/443 usually covers it, but if any server sits in a different ACG, or a rule was narrowed during hardening, that target alone will flap.
4. Make the endpoint cheap. If the health check path runs a database query or renders a full page, it will fail under load before the app is genuinely unhealthy. Replace it with a dedicated /healthz that returns 200 quickly and does not depend on downstream systems, and align thresholds, interval, and timeout with the real recovery time.
5. Check connection handling. Idle timeouts and keep-alive mismatches between the load balancer and the web server produce resets that look exactly like flapping.

Diagnosis to state explicitly. The evidence points at a health check design flaw amplified by traffic - an expensive or redirect-prone probe with a timeout that is too tight - rather than a VPC routing problem.

Root cause: the response was cacheable and the cache key did not include identity. The CDN will cache a GET response it considers static, typically keyed on host plus path, ignoring the session cookie unless told otherwise. If the storefront sends no Cache-Control and no Vary: Cookie, an authenticated HTML page for user A is stored and then served to user B. This is a security incident, not a performance issue - treat it as a data exposure: purge the cache immediately, note the exposure window, and check whether the CDN or access logs can bound which users were affected.

Policy design.

• Authenticated HTML is never shared-cacheable. Send Cache-Control: private, no-store (or at minimum private, max-age=0, must-revalidate) on any response that is rendered for a logged-in user. private is the key directive; no-store is stronger and appropriate for account and cart pages.
• Split what is static from what is dynamic. Put assets on a separate hostname or path prefix (/static/...) with long max-age and content-hashed filenames, so you get CDN benefit without ever caching the shell HTML.
• If HTML must be cached, make the key explicit. Configure the CDN to include the session cookie in the cache key, or bypass cache when an auth cookie is present. This costs hit ratio but is safe.
• Make the origin the authority. Origin headers should be set deliberately, not left at framework defaults, and purged by API as part of every deploy.

Verification. After the change, test with two distinct sessions: log in as A, fetch a page, log in as B, and confirm a cache MISS at origin for B's request. That test belongs in the release checklist.

Peering cannot be built on this pair as specified. VPC peering requires non-overlapping address ranges, because after peering each VPC must still have an unambiguous route to a destination. With both sides using 10.10.0.0/16, 10.10.1.20 in the portal's VPC and 10.10.1.20 in the storefront's VPC are the same address from a routing standpoint. You cannot resolve that with NAT inside a peering relationship, and forcing it produces silent misrouting - traffic intended for the partner lands on a storefront host. Reject the request rather than accepting a configuration that will fail unpredictably in production.

Options, with trade-offs.

1. Re-IP the partner portal (recommended). It is new and presumably lightly loaded, so move it to an unused range (for example 10.20.0.0/16) while it still has few dependencies, then peer normally. Cheapest long-term, highest one-time effort.
2. Do not peer at all - expose a narrow interface. Put the internal API behind an internal load balancer, and let the partner reach it through a deliberate path: an IPsec VPN between the two VPCs (or a shared services VPC that both connect to) with firewall rules permitting only the specific API port from the partner's range. This also gives you logging and a control point, which raw peering does not.
3. Small shared-services VPC. Terminate the integration in a dedicated VPC with non-overlapping addressing and let both sides talk only to that.

Whatever you choose, set the contract first: which endpoints, which direction, which ports, and who owns the firewall rules. Peering is often requested as a shortcut to skip that conversation; the conversation is the valuable part.