Hardcoded Secrets, Zero-Downtime Rotation, and a Legacy Java App
A payments company runs 220 microservices on Kubernetes (400 pods) and 60 Java services on virtual machines. It has 1,800 secrets: database passwords, API keys, and certificates. A scan finds 900 secrets in Git history and 150 in Kubernetes environment variables. Database credentials are rotated every 45 days by a script; each rotation causes 10-15 minutes of HTTP 500 errors because pods cache database connections. The company has licensed CyberArk Secrets Manager (Conjur) but only 12 applications use it. A legacy Java application cannot be modified; it reads jdbc.properties at startup and requires a restart to pick up changes. Audit requires proof that no secret remains in Git and that database credentials rotate every 24 hours. The platform team has four engineers and cannot manually restart 400 pods on every rotation.
1. Design the target architecture for 24-hour database credential rotation with no user-visible downtime. Explain the key design choices.
Read the worked answer
The core problem is that rotating a single credential invalidates every cached connection at the same moment. The design should separate credential issuance from connection establishment.
Use Conjur dynamic secrets if the database supports short-lived users: the application authenticates to Conjur with its Kubernetes service account identity, receives a unique database credential, and opens a connection. The credential expires naturally, and there is no global rotation event. If dynamic secrets are not possible, use a two-account rotation pattern: each application has an active and standby database account, rotation updates the standby first, applications gradually reconnect, and the roles swap. This avoids a single point of failure.
On Kubernetes, remove secrets from environment variables and mount them as files or fetch them at runtime through the Conjur Kubernetes authenticator and a sidecar or init container. For Java services on VMs, use the Central Credential Provider or Conjur API at startup and on connection-pool refresh. Configure the connection pool to validate connections and retry authentication on failure.
The key choices are runtime retrieval, short-lived or dual credentials, and connection-pool behavior. A faster rotation script alone would only make the outage happen more often.
2. The legacy Java application cannot be modified and reads jdbc.properties at startup. What are the realistic options, and which would you recommend?
Read the worked answer
There are three realistic options. First, inject the secret into jdbc.properties at deploy time from Conjur or CCP, then restart the application on a schedule that matches rotation. This works without code changes, but rotation and restart must be coordinated, and the file must not be baked into an image or committed to Git. Second, use a credential proxy or secretless broker in front of the database so the application connects to a local endpoint and the proxy injects the current credential. This avoids application changes and removes the secret from the app entirely, but it adds a network hop and a component to operate. Third, leave the application on a longer rotation cycle with a documented exception, which is the weakest option because audit requires 24 hours.
I would recommend the proxy or secretless approach if the database protocol is supported, because it decouples rotation from application restarts. If that is not feasible, use deploy-time injection plus an automated rolling restart, and treat the restart as part of the rotation job rather than a manual step. The decision should be based on protocol support, latency tolerance, and operational capacity, not on convenience.
3. Audit asks for proof that no secret remains in Git and that rotation actually happens every 24 hours. What evidence and controls would you put in place?
Read the worked answer
First, rotate every secret found in Git history before cleaning the repository. Cleaning history without rotation only hides the exposure. Then purge the secrets from history with a tool such as git filter-repo or BFG, force-push, and require all clones to be re-cloned. Add pre-commit and CI scanning so a new secret is blocked before merge.
For evidence, keep three artifacts: a before-and-after secret scan report showing the repository is clean, rotation logs from Conjur or CPM showing the timestamp and target account for each 24-hour cycle, and application logs or synthetic transactions showing successful authentication after rotation. A dashboard that shows last rotation time per account and alerts when an account exceeds 24 hours is stronger than a monthly report.
Finally, verify that the application is actually consuming the vaulted secret, not a cached copy. A configuration drift check should confirm that no jdbc.properties file on disk contains a static password. The audit answer is not a policy statement; it is the combination of repository evidence, rotation telemetry, and runtime verification.