Dynamic database credentials for a tool that only reads its config at start-up
A BI team's reporting service connects to PostgreSQL through the database secrets engine mounted at database/. It currently uses one static account bi_reader whose password sits in a config file. The platform team wants dynamic credentials: role bi-ro with default_ttl=1h, max_ttl=24h. Constraints: the tool holds a pool of 30 connections, reads credentials from its config file only at process start, takes 4 minutes to restart, and runs 24/7 with a change freeze on business hours. PostgreSQL has max_connections=200, with 60 already consumed by other applications. Security wants credentials that are useless within an hour of compromise, plus evidence that the database is not quietly accumulating orphaned roles as leases expire. Four instances of the service run behind a load balancer.
1. Is `default_ttl=1h` compatible with the tool's start-up-only config read? Reason through what actually happens at the one-hour mark and give the options in order of preference.
Read the worked answer
No, not as-is - but the breakage is delayed and therefore easy to miss in testing.
What happens: Vault revokes the lease at one hour and the plugin drops the PostgreSQL role. It does not reach into the tool and kill live sessions; the 30 pooled connections keep working because they are already established. The failure surfaces on the first reconnect, pool recycle, or network blip after the hour - the tool tries to re-authenticate with a username and password that no longer exist. On a 24/7 service that means a random 03:00 outage whose cause is an hour in the past, in a change freeze.
Options, best first:
- Make the credential renewable at connect time. Run Vault Agent alongside the tool with a template rendering the username/password into the config file the tool reads, plus a reload command on change (or a driver that supports a credential provider). Then TTL can stay at 1h and rotation becomes invisible.
- Raise the TTL to exceed the recycle interval, e.g.
default_ttl=24h, max_ttl=24h. Safe only if you accept a 24h exposure window - which contradicts the security ask, so treat it as a stopgap. - Static role with rotation (
rotation_period) if the tool cannot be changed at all: Vault still owns and rotates the password, but the tool has the same reconnect problem, so this buys inventory control, not short-lived credentials.
Do not deploy option 2 and call it done; the pool masks the problem until it doesn't.
2. How many database roles will four instances actually create, and what do you do about the connection budget and orphaned roles?
Read the worked answer
Roles track logins, not connections. One credential fetched at start-up serves all 30 pooled connections, so four instances create four PostgreSQL roles per TTL window - not 120. If the code fetches a credential per connection, that assumption breaks and you will create 120 roles per hour; state the one-credential-per-process rule explicitly in the client design and assert it in a test.
That keeps the role count trivial against max_connections=200. The real risks are:
- Roles outliving their lease. If Vault cannot reach PostgreSQL at revocation time, the role is left behind. Check inventory with
vault list sys/leases/lookup/database/creds/bi-roand clean up excess withvault lease revoke -prefix database/creds/bi-ro(scoped to this role, never cluster-wide). - Connection pressure comes from pool size, not Vault. Four instances × 30 connections is 120 of the 200 budget before Vault is involved. Raise this with the DBAs before adding a fifth instance; dynamic credentials change who owns the password, not the pool arithmetic.
- Renewal vs re-issue. Renewing the lease keeps the same username and password; re-reading
database/creds/bi-romints a new credential and a new lease, leaking a role each cycle. Renew; re-read only after a real expiry.
Measure role count over a week before declaring success - a slow leak is invisible in a one-hour test.
3. The tool is being decommissioned in three months. Argue for or against doing this migration now.
Read the worked answer
Against, with conditions. Three months is shorter than the cycle you need to prove the fix works: you want at least one month-end, one on-call rotation, and one unplanned reconnect to know the rotation path is real. A migration that lands a week before decommissioning buys almost no security - the static password is exposed for a few weeks less - while risking a 24/7 outage in a change freeze.
For, narrowly: the static bi_reader password is in a config file on four hosts today, and that is exactly the credential that gets copied into a wiki. A cheap intermediate step gets most of the benefit with none of the integration risk: rotate the existing static password on a schedule, or use a static database role so Vault owns it, and cut the config file out of the picture.
Decide on these questions: does the tool sit on any path the compliance scope cares about (then rotate now, in the simplest form that works); can the three months absorb a rollback (if not, don't start); and is the replacement tool going to use dynamic credentials from day one (then spend the three months on the replacement's design instead, and make dynamic credentials a launch requirement).
Write the decision down either way - the next tool will face the same question.