Vault Secret Retrieval on Every Function Invocation Causes 429s at Peak
Your team runs orders-fn, a Java 17 function in OCI Functions (application orders-api, 256 MB memory, 30-second timeout, no provisioned concurrency) that serves a checkout API at roughly 40 requests per second at peak. On every invocation it calls OCI Vault over a service gateway to fetch the secret orders-db-password, and the code builds a new SecretBundleClient before opening a connection to Autonomous Transaction Processing. p95 latency is 210 ms at 1 rps but 2.4 s at 40 rps. During peaks the function logs TooManyRequests (HTTP 429) responses from the Vault endpoint, followed by read timeouts that consume most of the 30-second budget. The password rotates every 30 days. Warm function instances are observed to survive several minutes between invocations. The team's first proposal is to raise memory to 1024 MB and wrap the Vault call in retries with exponential backoff. Monitoring shows no CPU or memory saturation at 256 MB.
1. Explain why the 429s and the latency curve appear, and evaluate the team's mitigation proposal.
Read the worked answer
Root cause chain. The Vault call is in the hot path, so Vault request volume equals invocation volume plus retries, not a fixed startup cost. At 40 rps with bursty concurrency, dozens of function instances each perform their own GetSecretBundle, each paying TLS setup, request signing and an IAM authorization check. OCI Vault enforces per-tenancy request-rate quotas on secret retrieval; once the quota is exhausted the service returns 429, and every retry multiplies the pressure. The 2.4 s p95 is consistent with SDK retry/backoff inside the call, and the later timeouts are those retries eating the 30-second function budget.
Why the proposal fails. Raising memory buys CPU share, but monitoring already shows no saturation - the bottleneck is outbound I/O and a remote quota, not local compute. Adding retries without jitter and without a circuit breaker increases the number of calls made precisely when the quota is already exceeded, converting an error storm into a timeout storm and burning the function timeout budget.
What the evidence tells you. The 30-day rotation cadence and the multi-minute instance lifetime are the important facts: a warm instance can safely hold the secret far longer than one invocation needs. The design error is treating a slowly changing secret as per-request data.
2. Design the secret-retrieval path inside the function so rotation still takes effect, and justify the numbers.
Read the worked answer
Design.
- Retrieve the bundle once per warm instance, in a static/singleton holder, not per invocation:
volatile SecretBundle cached; volatile Instant loadedAt;. - Reuse one
VaultsClient/SecretsClient(singleton), built with the function's resource principal - client construction is the expensive part after the network call. - Apply a TTL, e.g. 10 minutes, plus event-driven invalidation: if the database rejects authentication (ORA-01017) or Vault returns 401/404, drop the cache and refresh once, then fail fast rather than loop.
- Combine with a rotation runbook: rotate the ATP password first, then write the new secret version, then wait one TTL before retiring the old credential - this bounds the window in which an instance can present a stale password.
Justification of the numbers. With ~50 concurrent instances and a 10-minute TTL, steady-state Vault traffic is on the order of 5 calls per minute instead of 2,400 - three orders of magnitude below the ceiling that produced the 429s, and comfortably inside the quota even with autoscaling headroom. A 10-minute TTL means a rotation is fully propagated well within any reasonable incident window, while a 30-day rotation cadence means the TTL adds no meaningful operational risk. Cold starts remain the only guaranteed Vault call, which is the correct place to pay it.
Guardrails. Add a circuit breaker around Vault so a quota event degrades to cached credentials rather than to timeouts, and emit a metric when the refresh path is hit so silent staleness is visible.
3. The security team asks whether the password can be removed from the architecture entirely. What is your recommendation and what are the trade-offs?
Read the worked answer
Recommendation: yes, via IAM database authentication to Autonomous Transaction Processing. The function's resource principal (function OCID in a dynamic group) can obtain a short-lived database token, and the database user is mapped to that IAM principal, so no password exists to store, rotate or leak. The function then retrieves no secret at all, which removes the Vault call from the hot path by construction rather than by caching.
Consequences to plan for.
- Connection pooling changes shape. The token is short-lived, so pool entries outlive their credential. You must either refresh the pool on token renewal or use a pool implementation that re-authenticates; a long-lived HikariCP pool created once at cold start will start failing mid-life.
- Blast radius of IAM. Authorization now depends on the dynamic group rule and policy being correct; a mis-scoped rule breaks the function in a way that looks like a database outage, not an auth error.
- Cold start cost shifts. The token acquisition still happens at cold start, so first-invocation latency remains, though it is per-instance rather than per-request.
- Operational ownership. DBAs lose the familiar password-rotation lever, so the rotation runbook must be replaced by a token/pool runbook.
Judgement call. If the platform is already standardising on resource principals and can absorb the pool lifecycle work, IAM auth is the better long-term answer. If the pool is complex or shared, ship the TTL cache first - it is a small, reversible change that fixes the incident - and treat IAM auth as a separate, tested migration.