Dynamic Database Credentials Expire Behind a Connection Pool
A payments platform runs 140 services on EKS and uses Vault 1.15 with the database secrets engine for PostgreSQL. Each service gets a 1-hour dynamic credential through AppRole; the secret_id is stored in a Kubernetes Secret and mounted as an environment variable. Vault tokens have a 30-minute TTL. The Java services use HikariCP with maxLifetime set to 90 minutes and capture the username and password once at pod start. After a recent Vault upgrade, about 5% of requests fail every hour with password authentication failed for user v-app-...; Vault audit logs show database lease renewal requests succeeding but token renewal denied after 30 minutes. On-call restarts pods to recover. Security also wants to remove long-lived Secret objects and reduce blast radius. The platform must keep the change inside the existing EKS and Vault clusters.
1. Explain the failure chain and the immediate mitigation that does not require redesigning authentication.
Read the worked answer
The failure chain has two clocks. The database credential lease is one hour, but the Vault token expires after 30 minutes. The application does not renew through Vault Agent; it holds the initial username and password. When the database lease reaches rotation, new connections need the new password, but the pool still presents the old one. Existing PostgreSQL connections may survive a password change, which explains why only some requests fail rather than all traffic. Restarting pods works because they fetch a fresh credential, but it recurs every hour.
Immediate mitigation: lower HikariCP maxLifetime below the credential TTL, for example 45 minutes, and shorten idleTimeout so connections are recycled before rotation. Increase the Vault token TTL above the database lease TTL only as a temporary measure, and ensure the token has permission to renew the database lease. Do not simply lengthen the database TTL to 24 hours; that reduces rotation benefits and hides the design flaw. Roll out the pool change gradually and watch authentication errors per service.
2. Design the target authentication and credential-delivery pattern using Kubernetes and Vault.
Read the worked answer
Replace AppRole secret_id with Vault Kubernetes auth. Each service runs under a dedicated Kubernetes service account; a Vault role binds that service account and namespace to a narrow policy that can read only its database credential path. The Vault Agent injector, or a CSI volume, delivers the credential to a memory-backed volume rather than an environment variable. The application reads the lease file and reloads credentials before expiry.
The database role should issue a one-hour lease, and the application pool should recycle connections at roughly 45 minutes. Vault Agent authenticates with the pod's service-account token, renews its own token, and renews or re-fetches the database lease. Remove the Kubernetes Secret containing the static secret_id. Scope the database creation statements to the minimum tables or schema the service needs. This removes long-lived credentials and makes each pod's identity the basis for access, so revoking a service account or Vault role cleanly cuts off that workload.
3. What happens during a 20-minute Vault outage, and how should the platform behave?
Read the worked answer
During a Vault outage, existing database leases remain valid until their TTL, but the agent cannot renew them or fetch replacements. If the outage is shorter than the remaining lease, healthy connections continue. If it extends past lease expiry, the database will reject new connections using the expired credential. Restarting all pods during the outage makes the incident worse because they cannot authenticate to Vault and will crash-loop.
Design for graceful degradation: keep database TTLs long enough to cover a plausible outage, but not so long that revocation is ineffective; alert when lease renewal failures begin; and use readiness probes that distinguish 'cannot renew yet' from 'credential expired'. The application should retry credential fetch with backoff and keep existing connections until they fail. Run Vault with a highly available storage backend and test failover. After recovery, do not mass-restart; let the agent re-authenticate and rotate credentials while monitoring error rates.