6,400 Secrets, 3,100 Vaulted: Where Do You Start When the Freeze Ends?
Meridian Retail Bank has 212 applications and roughly 6,400 secrets in scope: database credentials, API keys, TLS private keys, and webhook signing secrets. About 3,100 sit in the secrets manager. The rest live in Kubernetes manifests, Jenkinsfiles, .env files on build agents, and one shared spreadsheet on a jump host. Seventy-eight percent of vaulted secrets are static and have never been rotated, and a single database credential is shared by fourteen applications. Sixty TLS certificates expire within 45 days and there is no reliable inventory of where they terminate. In the past year two secrets leaked into public repositories; the first took nine days to revoke because nobody could identify which teams consumed it. Developers resist onboarding because it requires code changes and a redeploy, and the release train is frozen for the next six weeks.
1. Rank the 3,300 unmanaged secrets for onboarding. What is your scoring model, and what do you do with the two previously leaked secrets?
Read the worked answer
The leaked secrets are not onboarding candidates - they are incidents. Treat both as compromised regardless of whether they are in the vault now. Revoke and reissue first, then find every consumer by searching build logs, repositories, and egress traffic. A secret that reached a public repository must be assumed harvested within minutes; the nine-day revocation window is the real finding.
Score the rest on three factors: blast radius (how many systems and what privilege), exposure (public repo, world-readable jump host, build agent readable by any pipeline), and time-to-fix (can it be injected without a code change).
That ordering puts: the 14-application shared database credential first, then anything on the jump host and build agents, then credentials for externally reachable services, then application-local secrets. Long-tail per-app secrets with single consumers go last, in batches by team.
Use the six-week freeze deliberately. You cannot redeploy, so build the integrations - sidecar or CSI-driver injection for Kubernetes, external-secrets operator, Jenkins credential binding - and pilot on a non-frozen application. Onboarding secrets into a vault nobody can consume is inventory theater; the deliverable at freeze end should be a pipeline that pulls at run time.
2. One database credential serves fourteen applications. What is the migration path to per-application or dynamic credentials, and what breaks first?
Read the worked answer
Step 1 - build the consumer map. You cannot retire a shared credential you cannot enumerate. Use database session logs, connection source IPs, and application inventory to attribute each connection, and confirm with each team. The nine-day revocation is the cost of skipping this step.
Step 2 - create per-application database accounts with least privilege. One role per app, scoped to the schemas and verbs it actually uses. Migrate one application at a time with the shared credential kept alive but unused as rollback.
Step 3 - decide static versus dynamic per consumer. Dynamic short-lived credentials are the goal, but connection pooling is the thing that breaks: pools hold connections open, so a 15-minute TTL produces constant churn and connection storms. Either front the database with a proxy that brokers credentials, or use longer TTLs with pool recycling. Do the arithmetic against the pool's max lifetime before choosing a TTL.
Step 4 - delete the shared credential and verify no sessions remain.
Track a single metric: applications holding a credential that another application also holds. Target zero, with exception dates.
3. Sixty certificates expire within 45 days and nobody knows where they terminate. What is your plan, and what do you do about clients that pin certificates?
Read the worked answer
Build the inventory from observation, not from a spreadsheet. Scan the external IP ranges and internal load balancers on 443, read the presented certificate chain, and cross-reference certificate transparency logs for externally issued certs. That gives you endpoint, issuer, and expiry - enough to act. Owner attribution comes second, from the load balancer or service catalog.
Triage the 60: external and payment-path certificates first, internal service-to-service second. For anything with an automated issuer, move to ACME-style renewal now so this never recurs. For internal CA-issued certificates, automate CSR generation and deployment through the pipeline.
Certificate pinning is the trap. Mobile clients and partner integrations that pin a certificate or public key will fail hard on renewal, and the failure appears at the client, not the server. Identify pinned endpoints, keep the same key pair where pinning is by key, and ship the new chain to clients before cutover with a dual-trust window.
Monitoring: expiry alerts at 30, 14, and 7 days routed to the owning team, not to a shared mailbox. A certificate expiry is a self-inflicted outage with a known date - the only acceptable failure mode is one you chose.