Key Vault Lockdown Breaks the Secrets Store CSI Driver

Why the fix is wrong: Key Vault Contributor is a management-plane role. It permits reading the vault's configuration and access metadata, not reading secret values. The data plane needs Key Vault Secrets User, which the identity already has - so no role change is required and none would help.

What 403 tells you: a network-path failure looks different. If DNS pointed at the private endpoint and the NSG blocked it, you would see a connection timeout. A fast 403 means the request reached the vault's public endpoint, which rejects it because public network access is disabled. So resolution is still returning the public IP.

Root cause: the privatelink.vaultcore.azure.net zone is not linked to the cluster VNet. Private DNS zone links are not transitive; the operations VNet link does nothing for pods in the cluster VNet.

Fix:

• Link the private zone to the cluster VNet (registration off), or attach a private DNS zone group to the private endpoint so the link is managed as part of the endpoint.
• Verify from inside a pod that <vault>.vault.azure.net resolves to the 10.20.4.x private IP, not a public address.
• Confirm the app subnet's NSG permits outbound 443 to 10.20.4.0/24, and that the CSI driver's SecretProviderClass still names the correct client ID and service account.

Several independent mechanisms have to line up:

1. Versionless secret URIs in the SecretProviderClass (secret/name rather than secret/name/<version>). With a pinned version the driver keeps serving the old value forever, and rotation silently does nothing.
2. Overlap the versions in Key Vault. Write the new secret as a new version before revoking the old one, and make sure the downstream system (database, third-party API) accepts both during the window. Rotation failures are almost always an ordering problem, not a driver problem.
3. Understand the refresh contract. The CSI driver polls Key Vault on a rotation interval and rewrites the mounted files. Environment variables are set at container start and are never refreshed - if the app reads from env vars it needs a restart regardless.
4. Make the app re-read. Either the application watches the mounted file, or you add a reloader that triggers a rolling restart on secret change. A rolling restart, not a delete, keeps availability.
5. Token refresh is the driver's job via the federated identity; if pods start failing after ~24 hours, look at the federated credential subject/issuer mismatch rather than the secret.

The failure was a deployment-order and ownership problem, not a Key Vault problem, so fix it where it is created:

• Use private DNS zone groups on the private endpoint rather than hand-linking zones. The link then travels with the endpoint and cannot be forgotten, and it is removed cleanly when the endpoint is deleted.
• Deploy the network lockdown and the client configuration in one change (infrastructure as code), with a canary namespace in the cluster validated before the vault's public access is set to Disabled. Disabling public access first and fixing clients later is what caused the outage.
• Add an Azure Policy audit that flags private endpoints whose private DNS zone group is missing or whose zone is not linked to the consuming VNet, plus an alert on Key Vault 403 spikes from first-party clients.
• Document the resolution dependency: any workload with a custom DNS server must forward privatelink.* zones to 168.63.129.16 or to a resolver that can, otherwise the next cluster repeats this exactly.