The Container Image That Shipped a Secret
A SaaS team ships 40 microservices via CI/CD, deploying ~30 times a day. A developer hardcoded a database password into a service's source to 'unblock' a demo, then removed it two commits later. But the secret was baked into a container image that was pushed to the registry and deployed to production; the image is still running in 3 environments and cached in the registry. The same password is used by 6 other services and has never been rotated in 2 years. The team has no secret scanner in CI, and the registry has no retention policy. A junior engineer asks whether this is 'really a big deal since it was removed from the code.'
1. Is removing the secret from source sufficient? Explain the exposure.
Read the worked answer
No - removing it from source fixes nothing that already happened.
The secret's exposure surface:
- Git history: the commit is still in history unless rewritten; anyone with repo access can retrieve it.
- Container image: the secret is baked into a layer that is still running in 3 environments and cached in the registry. Image layers are immutable - deleting the source line doesn't change the deployed image.
- Shared credential: the same password protects 6 other services, so the blast radius is far larger than one app.
- Age: 2 years unrotated means any prior leak is also live.
Conclusion: the secret is live and deployed, not removed. The junior engineer's mental model ('gone from code = gone') is the classic mistake. The correct model: a secret is compromised the moment it's committed, and stays compromised until rotated and purged from every copy (history, images, registry cache).
2. What is the remediation sequence, and why does order matter?
Read the worked answer
Order matters because rotation is the only step that actually kills the exposure; everything else is cleanup.
- Rotate the credential first - issue a new password for all 6 services and deploy it via the secret manager. Until this happens, every other step is cosmetic; the old password still works.
- Purge copies: rewrite git history (and force-push/rotate any tokens that touched it), rebuild images without the secret, redeploy all 3 environments, and purge the registry cache (set a retention policy so old layers don't linger).
- Add prevention: secret scanning in CI (block on detection), pre-commit hooks, and a managed secret store so there's no reason to hardcode.
- Verify: confirm the old password no longer authenticates anywhere and no image contains it.
Why not purge first? Rewriting history and rebuilding images takes time; the live credential remains valid throughout. Rotate first, then clean. If you clean first and rotate later, you've spent hours with a working leaked password.
3. What systemic controls address the root cause so this recurs less?
Read the worked answer
Root cause is not 'a careless developer' - it's that hardcoding was the path of least resistance and nothing detected it.
- Make the right way easy: a managed secret store integrated with CI/CD, so pulling a secret is trivial and hardcoding is unnecessary.
- Detect at the earliest gate: secret scanning in pre-commit and CI, failing the build. Catching it at commit is orders of magnitude cheaper than at runtime.
- Registry hygiene: retention policy and image scanning so stale layers don't persist indefinitely.
- Credential hygiene: per-service credentials (not one password for 6 services), rotation schedule, and a register of secret age.
- Culture: the 'unblock the demo' pressure is real; give a sanctioned fast path (ephemeral dev secrets) so the shortcut isn't needed.
Exam-tested framing: treat leaked secrets as a process and detection problem. Blame doesn't scale; automated prevention at the commit gate and a low-friction secret store do.