Kubernetes Secret Migration With a Shared Namespace Identity
Arcadia Retail is moving 300 microservices to EKS. Today there are roughly 1,100 static credentials: 600 in the Jenkins credential store, 350 in Helm values committed to Git, and 150 hard-coded in Java properties files on VMs. The plan is to adopt CyberArk Secrets Manager as the system of record, with Secrets Hub syncing 400 RDS credentials to AWS Secrets Manager for Lambda consumers. In the pilot, the JWT authenticator was configured at namespace level, so all services in the namespace share one service account and the orders-api pod was able to read payments-db credentials. During the first automatic rotation the orders service held 42 pooled connections open and threw 900 errors in four minutes. Security wants 24-hour rotation; the DBA team wants 90 days. Five months remain before the VM estate is decommissioned.
1. Fix the pilot's blast radius problem. Explain the identity and entitlement model you would apply to all 300 services.
Read the worked answer
The pilot's flaw is that authentication was scoped to the namespace rather than to the workload. A namespace-level service account means every pod in that namespace presents the same identity, and the authorization model can therefore only grant at namespace granularity - so orders-api inheriting payments-db access is the expected outcome, not an accident.
Design:
- One Kubernetes service account per workload, one Secrets Manager host identity per workload. The authenticator must restrict the token's subject claim to the exact service account (namespace plus service account name), not a wildcard or the namespace alone. A token minted for any other workload is then rejected at authentication, before authorization is even consulted.
- Authorize at the secret layer, not the application layer. Each service's secrets live in their own branch of the policy; the host is granted read on that branch only. There is no shared 'apps' layer that every service can read.
- Separate read from rotate. Application identities get read; only the platform's rotation identities get update, so a compromised app cannot rewrite a credential it depends on.
- Deliver secrets as mounted files rather than environment variables. Environment variables leak through process listings, crash dumps, child processes, and debugging endpoints, and they cannot be refreshed without a restart.
The general rule: identity granularity determines the finest possible authorization granularity. Get per-workload identity first; entitlement scoping is impossible without it.
2. Reconcile the 24-hour versus 90-day rotation dispute and design a rotation approach that does not repeat the four-minute outage.
Read the worked answer
The outage was caused by rotation without a consumer-side refresh path: the password changed, the pool kept using the old value, and every new connection attempt failed. The rot interval was not the problem. Any interval produces this failure; a 90-day interval just makes it rarer and more surprising.
Recommendation:
- For the 400 RDS credentials, prefer short-lived dynamic database credentials over a single rotating static account. Ephemeral users with a TTL mean each new connection authenticates with a freshly minted credential, so there is no stale cached value to break. Configure the connection pool's maximum lifetime to be comfortably below the credential TTL, and cap idle connections.
- Where a static account must remain (vendor constraints, reporting users), use a dual-account pattern: two accounts alternate, rotation happens on the idle one, consumers fail over on authentication error, and both are never rotated in the same window.
- Set rotation to 24 hours for application credentials, since the compromise window is what the control buys. Accept 90 days only for break-glass and human-used accounts with vault-mediated check-out, where every use is recorded.
- Account for Secrets Hub sync latency. If rotation happens at 02:00 and the sync job runs hourly, consumers downstream of the synced copy see up to an hour of skew - size the rotation and sync windows accordingly and monitor sync failures as a production alert.
- Watch the first three cycles of each service with error-rate alarms before declaring the migration done.
3. The platform team asks whether to inject secrets with an init container or a refreshing sidecar/CSI volume. Which do you choose for a service whose pods live for weeks, and why?
Read the worked answer
Choose the refreshing delivery mechanism - a CSI-mounted volume or sidecar that re-reads the secret and updates the mounted file - not an init container alone.
Reasoning: an init container fetches the credential once, at pod start. With 24-hour rotation and pods that live for weeks, the mounted value goes stale and the pod either breaks at the next connection or keeps using a revoked credential until someone restarts it. That reproduces the pilot outage in slow motion and is far harder to diagnose because it looks intermittent and correlates with pod age rather than with a deploy.
A refreshing mechanism keeps the file current in place; the application only needs to re-read the file on new connections rather than cache it in memory forever. That is a small, well-understood application change.
Two supporting decisions:
- Keep the secret as a file, not an environment variable, so refresh is possible at all.
- Where the application genuinely cannot re-read a file, accept a restart-on-rotation pattern explicitly and automate it, rather than silently depending on stale values.
Init containers are acceptable for bootstrapping a one-time token or for short-lived jobs whose lifetime is shorter than the rotation interval. They are a poor fit for long-lived services under 24-hour rotation.