240 Live Secrets, a Four-Month Deadline, and a Shared Deploy User
Orbit Logistics has 312 Git repositories; a scan found 1,900 live credentials in commit history and 240 in current code. The platform team selected HashiCorp Vault and has 4 months before its SOC 2 audit. Applications run on 90 EC2 instances, 40 ECS services, and a Jenkins instance with 2,300 job definitions, many of which have no identifiable owner. The database team refuses dynamic database credentials because connection pooling in a legacy Java application breaks on rotation, and a 2023 rotation attempt caused a 4-hour outage. Three teams currently share a single 'svc_deploy' IAM user with long-lived access keys. The CTO wants to know whether 4 months is enough and what 'done' looks like.
1. Sequence the migration. What is fixed in week one, what waits until after the audit, and how do you avoid causing the outage the CTO is afraid of?
Read the worked answer
Week one is about exposure, not architecture. Validity first: a secret in commit history that is still active is a current-code secret wearing a disguise. Deduplicate the 240 current-code findings and the 1,900 history findings by credential, test validity, and rank by (a) internet-reachable scope, (b) production blast radius, (c) whether the credential grants identity or IAM changes.
Fix in week one: any cloud key with administrative or broad data scope, any token reachable from outside, and any credential in a public or broadly shared repository. Rotation here is uncontroversial because the correct end state is revocation.
The shared 'svc_deploy' user is the trap. Do not rotate it first. Three teams depend on it and you cannot coordinate a rotation you cannot attribute. Days 1-3: create per-team identities with equivalent permissions, migrate each team with them watching, and only then disable the shared user. Rotating first produces exactly the 2023 outage and destroys the political capital for the rest of the programme.
After the audit: the 1,900 history findings that are no longer valid, secret-scanning gates in CI, and rewriting history where required.
Definition of done: no long-lived static cloud credentials in code or CI; secrets retrieved at runtime with an auditable identity; rotation demonstrated by a game day, not by a policy document.
2. The database team says dynamic credentials break pooling and cites a 4-hour outage. Is a static credential with automated rotation acceptable? What conditions would you attach?
Read the worked answer
Yes, conditionally. Refusing rotation entirely is not acceptable; refusing short-TTL dynamic credentials for one legacy application is a reasonable engineering position if it is bounded.
Conditions:
- Dual-account rotation with overlap. Maintain two accounts; rotate the inactive one, validate it, update the secret store, then let the application pick it up. Never disable the old password until the new one has been proven by a health check.
- Rotation cadence matched to pool and connection lifetime. If connections live for hours, a rotation that invalidates the old password in minutes will fail. Set the overlap window longer than the maximum connection lifetime, or require the pool to drain.
- Retry and reconnect logic verified in staging with production-shaped pooling, not in a lab.
- Rotation must be monitored and reversible. An automated rotation that fails silently at 02:00 is how you get the 2023 outage again.
- Restrict the static credential's reach - network policy to the app tier only, least-privilege grants, and logging on use.
- A review date and a remediation path - driver or pool upgrade that enables shorter TTLs.
The trade-off to state plainly: short TTLs reduce the window in which a stolen credential is useful, but they buy that with availability risk in a system that has already demonstrated it cannot tolerate churn. Take the longer TTL on the one system that needs it and buy back risk with network restriction and monitoring. Do not generalize the exception to systems that can handle rotation.
3. Jenkins has 2,300 job definitions, many unowned. How do you remove static secrets from CI without breaking pipelines nobody understands?
Read the worked answer
Treat CI as a class of secret to be eliminated rather than a place to store secrets better.
Eliminate the class where possible. Move cloud authentication to workload identity federation - CI presents an OIDC token and receives short-lived, per-job-scoped cloud credentials. Nothing long-lived exists to leak, and the token is bound to the specific pipeline. This resolves most of the 2,300 jobs without touching their logic.
For secrets that are not cloud credentials, use Vault with per-job identity (AppRole or Kubernetes auth) and scoped policies, so a job reads only what that job needs and the read is logged.
Avoid the breakage:
- Run in shadow mode first - inject the Vault-provided value alongside the existing one and log which jobs actually consumed it. Pipelines fail on missing variables, not on extra ones.
- Measure coverage as percentage of jobs with zero static secrets, not percentage of secrets migrated. The first metric cannot be gamed by moving a secret.
- For unowned jobs: notify, then disable after a defined window. If a job has not executed in 90 days, archive it rather than migrating it. Migrating abandoned jobs is how a four-month programme becomes a twelve-month one.
- Keep one documented break-glass path, monitored, so an on-call engineer is never tempted to re-introduce a static key at 03:00.
The real risk in CI is not the 240 findings; it is that fixing them by centralizing them into a Jenkins credential store leaves 2,300 jobs able to read secrets they do not need.